[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:13 dist-test-slave-9zdj krb5kdc[29113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:13 dist-test-slave-9zdj krb5kdc[29113](info): set up 2 sockets
Aug 07 08:42:13 dist-test-slave-9zdj krb5kdc[29113](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 08:42:14.777351 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:14 dist-test-slave-9zdj krb5kdc[29113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:14.942934 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:35337
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:35337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:15.045651 29127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:15.045912 29127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:15.045940 29127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:15.047319 29127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:15.047377 29127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:15.047392 29127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:15.047405 29127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:15.047418 29127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:15.052057 29127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:35337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:35337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:15.052688 29127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:15.053061 29127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:15.062422 29135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:15.063508 29133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:15.064083 29132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:15.064556 29127 server_base.cc:1034] running on GCE node
I20260807 08:42:15.064972 29127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:15.065582 29127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:15.066828 29127 hybrid_clock.cc:648] HybridClock initialized: now 1786092135066808 us; error 80 us; skew 500 ppm
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092135, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:15.070544 29127 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:15.071386 29127 webserver.cc:460] Webserver started at http://127.28.108.190:37803/ using document root <none> and password file <none>
I20260807 08:42:15.071782 29127 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:15.071853 29127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:15.072075 29127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:15.073544 29127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "14fb56da226c406d8d618f4d6b58ee38"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "6cc1146d5d36f217c674e2ec662e5dea"
server_key_iv: "bc9971f97385f708fa383631fcc59623"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.074090 29127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "14fb56da226c406d8d618f4d6b58ee38"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "6cc1146d5d36f217c674e2ec662e5dea"
server_key_iv: "bc9971f97385f708fa383631fcc59623"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.076223 29127 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 08:42:15.077494 29142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:15.077872 29127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:15.078027 29127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "14fb56da226c406d8d618f4d6b58ee38"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "6cc1146d5d36f217c674e2ec662e5dea"
server_key_iv: "bc9971f97385f708fa383631fcc59623"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.078141 29127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:15.095072 29127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:15.098289 29127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:15.098560 29127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:15.116048 29127 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:35337
I20260807 08:42:15.116418 29199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:35337 every 8 connection(s)
I20260807 08:42:15.116926 29127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:15.121455 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 29127
I20260807 08:42:15.121573 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:15.122285 29200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:15.122947 29106 external_mini_cluster.cc:1428] Setting key 46eb3e47771cd83dec5ec8c64c0477c0
I20260807 08:42:15.128589 29200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38: Bootstrap starting.
I20260807 08:42:15.130604 29200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:15.132149 29200 log.cc:824] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:15.133716 29200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38: No bootstrap required, opened a new log
I20260807 08:42:15.138348 29200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fb56da226c406d8d618f4d6b58ee38" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35337 } }
I20260807 08:42:15.138549 29200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:15.138616 29200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14fb56da226c406d8d618f4d6b58ee38, State: Initialized, Role: FOLLOWER
I20260807 08:42:15.138854 29200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fb56da226c406d8d618f4d6b58ee38" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35337 } }
I20260807 08:42:15.138983 29200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:15.139035 29200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:15.139091 29200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:15.140332 29200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fb56da226c406d8d618f4d6b58ee38" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35337 } }
I20260807 08:42:15.140597 29200 leader_election.cc:302] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14fb56da226c406d8d618f4d6b58ee38; no voters:
I20260807 08:42:15.140903 29200 leader_election.cc:288] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:15.141059 29205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:15.141445 29205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [term 1 LEADER]: Becoming Leader. State: Replica: 14fb56da226c406d8d618f4d6b58ee38, State: Running, Role: LEADER
I20260807 08:42:15.141507 29200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:15.141762 29205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fb56da226c406d8d618f4d6b58ee38" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35337 } }
I20260807 08:42:15.143198 29209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14fb56da226c406d8d618f4d6b58ee38. Latest consensus state: current_term: 1 leader_uuid: "14fb56da226c406d8d618f4d6b58ee38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fb56da226c406d8d618f4d6b58ee38" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35337 } } }
I20260807 08:42:15.143406 29209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:15.144349 29211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:15.143198 29208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14fb56da226c406d8d618f4d6b58ee38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fb56da226c406d8d618f4d6b58ee38" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35337 } } }
I20260807 08:42:15.145062 29208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38 [sys.catalog]: This master's current role is: LEADER
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:15.148890 29211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:15.153271 29211 catalog_manager.cc:1347] Generated new cluster ID: 6ecdb2b3482a49af99264ade318e5c25
I20260807 08:42:15.153343 29211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:15.153453 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:15.126667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.28.108.190:35337)
0807 08:42:15.127160 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:15.127174 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:15.127348 (+ 174us) server_negotiation.cc:407] Connection header received
0807 08:42:15.132142 (+ 4794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:15.132154 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:15.132291 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:15.132535 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:15.135462 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.137643 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.139419 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.140184 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.140330 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:15.147054 (+ 6724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:15.147060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:15.147060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:15.147076 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:15.150634 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.151380 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.151384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.151384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.151483 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.151886 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.151887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.151888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.152229 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:15.152429 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:15.152904 (+ 475us) server_negotiation.cc:299] Negotiation successful
0807 08:42:15.153110 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":166,"threads_started":1}
I20260807 08:42:15.161458 29211 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:15.162245 29211 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:15.164710 29211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14fb56da226c406d8d618f4d6b58ee38: Generated new TSK 0
I20260807 08:42:15.164958 29211 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:42:15.210816 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35337
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:15.332749 29230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:15.333091 29230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:15.333145 29230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:15.334760 29230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:15.334846 29230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:15.334896 29230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:15.336827 29230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:15.337224 29230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:15.337512 29230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:15.346300 29236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:15.346396 29237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:15.346661 29230 server_base.cc:1034] running on GCE node
W20260807 08:42:15.346716 29239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:15.346982 29230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:15.347301 29230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:15.348496 29230 hybrid_clock.cc:648] HybridClock initialized: now 1786092135348458 us; error 58 us; skew 500 ppm
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092135, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:15.352507 29230 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:42:15.353180 29230 webserver.cc:460] Webserver started at http://127.28.108.129:37441/ using document root <none> and password file <none>
I20260807 08:42:15.353497 29230 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:15.353556 29230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:15.353778 29230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:15.355284 29230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "a8ca7453540e45e9b382ae09d30328f1"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "f7c870c08b4d33183dbbbd0004a4958e"
server_key_iv: "1911e83706b9258311106c238526ed73"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.355764 29230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "a8ca7453540e45e9b382ae09d30328f1"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "f7c870c08b4d33183dbbbd0004a4958e"
server_key_iv: "1911e83706b9258311106c238526ed73"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.358007 29230 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 08:42:15.359359 29247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:15.359661 29230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:15.359758 29230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "a8ca7453540e45e9b382ae09d30328f1"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "f7c870c08b4d33183dbbbd0004a4958e"
server_key_iv: "1911e83706b9258311106c238526ed73"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.359918 29230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:15.412815 29230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:15.415822 29230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:15.416054 29230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:15.416550 29230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:15.417039 29230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:15.417143 29230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:15.417182 29230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:15.417213 29230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:15.434821 29230 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:35137
I20260807 08:42:15.434927 29368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:35137 every 8 connection(s)
I20260807 08:42:15.435415 29230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:42:15.440922 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 29230
I20260807 08:42:15.441061 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:15.441360 29106 external_mini_cluster.cc:1428] Setting key dde25aeaa16719321791972a2e8ebfa4
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092135, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:15.448179 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:15.436569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:33991 (local address 127.28.108.190:35337)
0807 08:42:15.436644 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:15.436647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:15.436991 (+ 344us) server_negotiation.cc:407] Connection header received
0807 08:42:15.437670 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:15.437672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:15.437686 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:15.437770 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:15.439039 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.439652 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.440993 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.441314 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.441409 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:15.444084 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:15.444086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:15.444087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:15.444095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:15.446010 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.446784 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.446785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.446786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.446861 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.447226 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.447227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.447228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.447401 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:15.447475 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:15.447938 (+ 463us) server_negotiation.cc:299] Negotiation successful
0807 08:42:15.448053 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 08:42:15.448372 29371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:15.436546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35337 (local address 127.28.108.129:33991)
0807 08:42:15.436800 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:15.436847 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:15.437505 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:15.437852 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:15.437856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:15.438695 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:15.438919 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:15.438925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.439888 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.439889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:15.440847 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:15.440850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.441418 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.441420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:15.441566 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:15.442341 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:15.442350 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:15.443945 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:15.446126 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:15.446156 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:15.446158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:15.446681 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:15.446975 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:15.446976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:15.446976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:15.447140 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:15.447488 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:15.447495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:15.447689 (+ 194us) client_negotiation.cc:769] Sending connection context
0807 08:42:15.447893 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 08:42:15.448103 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":124,"threads_started":1}
I20260807 08:42:15.449108 29369 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35337
I20260807 08:42:15.449236 29369 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:15.449493 29369 heartbeater.cc:511] Master 127.28.108.190:35337 requested a full tablet report, sending...
I20260807 08:42:15.450733 29161 ts_manager.cc:194] Registered new tserver with Master: a8ca7453540e45e9b382ae09d30328f1 (127.28.108.129:35137)
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
I20260807 08:42:15.453863 29161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:33991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:42:15.478605 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35337
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:15.653313 29374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:15.653750 29374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:15.653810 29374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:15.656229 29374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:15.656340 29374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:15.656416 29374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:15.659713 29374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:15.660087 29374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:15.660485 29374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:15.668393 29382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:15.668419 29380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:15.668560 29379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:15.669099 29374 server_base.cc:1034] running on GCE node
I20260807 08:42:15.669391 29374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:15.669785 29374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:15.671007 29374 hybrid_clock.cc:648] HybridClock initialized: now 1786092135670991 us; error 30 us; skew 500 ppm
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092135, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:15.675894 29374 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:42:15.676481 29374 webserver.cc:460] Webserver started at http://127.28.108.130:41165/ using document root <none> and password file <none>
I20260807 08:42:15.676736 29374 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:15.676817 29374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:15.676980 29374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:15.678537 29374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "0e093aed59b14f5a80deb3efea1811ab"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "dae5091d2a763d4cd8609f4eb3b323f4"
server_key_iv: "d3299d9824d4d781340f77cdffc12111"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.679005 29374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "0e093aed59b14f5a80deb3efea1811ab"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "dae5091d2a763d4cd8609f4eb3b323f4"
server_key_iv: "d3299d9824d4d781340f77cdffc12111"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.680979 29374 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260807 08:42:15.682353 29390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:15.682624 29374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:42:15.682859 29374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "0e093aed59b14f5a80deb3efea1811ab"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "dae5091d2a763d4cd8609f4eb3b323f4"
server_key_iv: "d3299d9824d4d781340f77cdffc12111"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.682967 29374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:15.699036 29374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:15.703037 29374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:15.703375 29374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:15.704082 29374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:15.704698 29374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:15.704757 29374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:15.704797 29374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:15.704821 29374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:15.716840 29374 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:35155
I20260807 08:42:15.716980 29508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:35155 every 8 connection(s)
I20260807 08:42:15.717427 29374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:15.718550 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 29374
I20260807 08:42:15.718650 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:42:15.718840 29106 external_mini_cluster.cc:1428] Setting key f0cf2337005c1766f24ab564999909de
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092135, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
I20260807 08:42:15.731379 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:15.718494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:52405 (local address 127.28.108.190:35337)
0807 08:42:15.718576 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:15.718579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:15.719043 (+ 464us) server_negotiation.cc:407] Connection header received
0807 08:42:15.720130 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:15.720132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:15.720145 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:15.720222 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:15.721927 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.722577 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.724409 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.724628 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.724700 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:15.727735 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:15.727737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:15.727737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:15.727747 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:15.729605 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.730250 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.730252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.730252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.730318 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.730609 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.730610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.730611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.730788 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:15.730868 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:15.731173 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 08:42:15.731281 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 08:42:15.731760 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:15.718451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35337 (local address 127.28.108.130:52405)
0807 08:42:15.718852 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:15.718898 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:15.719984 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:15.720295 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:15.720299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:15.721081 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:15.721587 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:15.721598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.722759 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.722760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:15.724273 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:15.724278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.724754 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.724755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:15.724934 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:15.725585 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:15.725598 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:15.727612 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:15.729815 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:15.729824 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:15.729826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:15.730096 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:15.730396 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:15.730397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:15.730397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:15.730532 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:15.730868 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:15.730877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:15.731032 (+ 155us) client_negotiation.cc:769] Sending connection context
0807 08:42:15.731148 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 08:42:15.731546 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":119,"threads_started":1}
I20260807 08:42:15.732439 29509 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35337
I20260807 08:42:15.732659 29509 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:15.732929 29509 heartbeater.cc:511] Master 127.28.108.190:35337 requested a full tablet report, sending...
I20260807 08:42:15.733546 29161 ts_manager.cc:194] Registered new tserver with Master: 0e093aed59b14f5a80deb3efea1811ab (127.28.108.130:35155)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
I20260807 08:42:15.734059 29161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:52405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:42:15.754349 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35337
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:15.852499 29514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:15.852756 29514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:15.852787 29514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:15.854197 29514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:15.854246 29514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:15.854278 29514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:15.855911 29514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:15.856133 29514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:15.856360 29514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:15.862779 29522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:15.862808 29520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:15.862823 29519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:15.863173 29514 server_base.cc:1034] running on GCE node
I20260807 08:42:15.863547 29514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:15.863837 29514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:15.865049 29514 hybrid_clock.cc:648] HybridClock initialized: now 1786092135865030 us; error 31 us; skew 500 ppm
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092135, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:15.867946 29514 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:42:15.868429 29514 webserver.cc:460] Webserver started at http://127.28.108.131:39295/ using document root <none> and password file <none>
I20260807 08:42:15.868691 29514 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:15.868739 29514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:15.868919 29514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:15.870191 29514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "527ee48fe38c4b7c81fca28994ed85fa"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "e4eab77402374401b980007f9fedd5a9"
server_key_iv: "fb96a9ffdf944d830f831b286b599639"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.870610 29514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "527ee48fe38c4b7c81fca28994ed85fa"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "e4eab77402374401b980007f9fedd5a9"
server_key_iv: "fb96a9ffdf944d830f831b286b599639"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.872236 29514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260807 08:42:15.873459 29530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:15.873922 29514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:15.874032 29514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "527ee48fe38c4b7c81fca28994ed85fa"
format_stamp: "Formatted at 2026-08-07 08:42:15 on dist-test-slave-9zdj"
server_key: "e4eab77402374401b980007f9fedd5a9"
server_key_iv: "fb96a9ffdf944d830f831b286b599639"
server_key_version: "encryptionkey@0"
I20260807 08:42:15.874105 29514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:15.887009 29514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:15.889971 29514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:15.890108 29514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:15.890334 29514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:15.890772 29514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:15.890818 29514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:15.890844 29514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:15.890858 29514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:15.898631 29514 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:45645
I20260807 08:42:15.898675 29655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:45645 every 8 connection(s)
I20260807 08:42:15.899093 29514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:15.901536 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 29514
I20260807 08:42:15.901692 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:15.901885 29106 external_mini_cluster.cc:1428] Setting key cec09d5e281d6e2b93aa2a55b5c7ff83
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092135, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:15.909745 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:15.900057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:50359 (local address 127.28.108.190:35337)
0807 08:42:15.900234 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:15.900237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:15.900546 (+ 309us) server_negotiation.cc:407] Connection header received
0807 08:42:15.901176 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:15.901177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:15.901192 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:15.901239 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:15.902545 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.903071 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.904136 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.904340 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.904427 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:15.906876 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:15.906879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:15.906879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:15.906887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:15.908202 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.908672 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.908674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.908674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.908736 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.909011 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.909012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.909013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.909141 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:15.909222 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:15.909522 (+ 300us) server_negotiation.cc:299] Negotiation successful
0807 08:42:15.909648 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 08:42:15.909987 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:15.900012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35337 (local address 127.28.108.131:50359)
0807 08:42:15.900394 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:15.900437 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:15.901053 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:15.901343 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:15.901346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:15.902157 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:15.902470 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:15.902477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.903187 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.903188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:15.904028 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:15.904030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.904439 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.904440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:15.904585 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:15.905151 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:15.905161 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:15.906771 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:15.908304 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:15.908313 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:15.908315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:15.908573 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:15.908816 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:15.908817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:15.908817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:15.908937 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:15.909213 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:15.909220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:15.909383 (+ 163us) client_negotiation.cc:769] Sending connection context
0807 08:42:15.909499 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 08:42:15.909701 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":112,"threads_started":1}
I20260807 08:42:15.910712 29656 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35337
I20260807 08:42:15.910812 29656 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:15.911026 29656 heartbeater.cc:511] Master 127.28.108.190:35337 requested a full tablet report, sending...
I20260807 08:42:15.911540 29161 ts_manager.cc:194] Registered new tserver with Master: 527ee48fe38c4b7c81fca28994ed85fa (127.28.108.131:45645)
I20260807 08:42:15.911972 29161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:50359
I20260807 08:42:15.914403 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092135, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092135, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
I20260807 08:42:15.943102 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:15.932904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.28.108.129:35137)
0807 08:42:15.933140 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:15.933145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:15.933169 (+ 24us) server_negotiation.cc:407] Connection header received
0807 08:42:15.933202 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:15.933204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:15.933220 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:15.933320 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:15.934317 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.935111 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.936077 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.936433 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.936492 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:15.939663 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:15.939675 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:15.939676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:15.939683 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:15.941531 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.942050 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.942054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.942055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.942132 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.942469 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.942470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.942470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.942638 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:15.942775 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:15.942907 (+ 132us) server_negotiation.cc:299] Negotiation successful
0807 08:42:15.942995 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
W20260807 08:42:15.943859 29276 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55720
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 08:42:15 dist-test-slave-9zdj krb5kdc[29113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
I20260807 08:42:15.972546 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:15.963725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.28.108.129:35137)
0807 08:42:15.963799 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:15.963801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:15.963901 (+ 100us) server_negotiation.cc:407] Connection header received
0807 08:42:15.964000 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:15.964001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:15.964014 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:15.964102 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:15.964898 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.965428 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.966294 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:15.966516 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:15.966599 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:15.969569 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:15.969571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:15.969572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:15.969586 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:15.971290 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.971691 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.971693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.971693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.971756 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:15.971985 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:15.971986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:15.971987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:15.972100 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:15.972204 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:15.972339 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 08:42:15.972434 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 08:42:15.973191 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 29230
I20260807 08:42:15.981827 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 29374
I20260807 08:42:15.987931 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 29514
I20260807 08:42:15.994431 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 29127
2026-08-07T08:42:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2463 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:16 dist-test-slave-9zdj krb5kdc[29672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:16 dist-test-slave-9zdj krb5kdc[29672](info): set up 2 sockets
Aug 07 08:42:16 dist-test-slave-9zdj krb5kdc[29672](info): commencing operation
krb5kdc: starting...
W20260807 08:42:17.132205 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:17 dist-test-slave-9zdj krb5kdc[29672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:17.293586 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42933
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:36835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:42933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:17.384848 29686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:17.385152 29686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:17.385196 29686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:17.386547 29686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:17.386632 29686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:17.386658 29686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:17.386693 29686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:17.386715 29686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:17.388290 29686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:36835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:42933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:17.388593 29686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:17.388847 29686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:17.394706 29695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:17.394729 29691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:17.394712 29693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:17.395027 29686 server_base.cc:1034] running on GCE node
I20260807 08:42:17.395407 29686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:17.395702 29686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:17.396867 29686 hybrid_clock.cc:648] HybridClock initialized: now 1786092137396834 us; error 44 us; skew 500 ppm
Aug 07 08:42:17 dist-test-slave-9zdj krb5kdc[29672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:17.400187 29686 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:17.400780 29686 webserver.cc:460] Webserver started at http://127.28.108.190:36021/ using document root <none> and password file <none>
I20260807 08:42:17.401036 29686 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:17.401114 29686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:17.401263 29686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:17.402647 29686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "aa7c7f9f382441a4bca718496957a083"
format_stamp: "Formatted at 2026-08-07 08:42:17 on dist-test-slave-9zdj"
server_key: "cee5f2bc373bbcdca5ad958b9620de65"
server_key_iv: "60f56e5c93d7fda7fe6b5b4b46109834"
server_key_version: "encryptionkey@0"
I20260807 08:42:17.403072 29686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "aa7c7f9f382441a4bca718496957a083"
format_stamp: "Formatted at 2026-08-07 08:42:17 on dist-test-slave-9zdj"
server_key: "cee5f2bc373bbcdca5ad958b9620de65"
server_key_iv: "60f56e5c93d7fda7fe6b5b4b46109834"
server_key_version: "encryptionkey@0"
I20260807 08:42:17.404731 29686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:42:17.405959 29702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:17.406196 29686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:17.406327 29686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "aa7c7f9f382441a4bca718496957a083"
format_stamp: "Formatted at 2026-08-07 08:42:17 on dist-test-slave-9zdj"
server_key: "cee5f2bc373bbcdca5ad958b9620de65"
server_key_iv: "60f56e5c93d7fda7fe6b5b4b46109834"
server_key_version: "encryptionkey@0"
I20260807 08:42:17.406411 29686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:17.416450 29686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:17.419400 29686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:17.419610 29686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:17.424623 29686 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:42933
I20260807 08:42:17.424669 29757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:42933 every 8 connection(s)
I20260807 08:42:17.425108 29686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:17.426515 29758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:17.429147 29758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083: Bootstrap starting.
I20260807 08:42:17.429876 29758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:17.430164 29758 log.cc:824] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:17.430873 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 29686
I20260807 08:42:17.430991 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:17.431021 29758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083: No bootstrap required, opened a new log
I20260807 08:42:17.431169 29106 external_mini_cluster.cc:1428] Setting key e4cfd8961d1196f68f87bfa1bc0af44f
I20260807 08:42:17.433470 29758 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7c7f9f382441a4bca718496957a083" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42933 } }
I20260807 08:42:17.433698 29758 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:17.433728 29758 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7c7f9f382441a4bca718496957a083, State: Initialized, Role: FOLLOWER
I20260807 08:42:17.433893 29758 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7c7f9f382441a4bca718496957a083" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42933 } }
I20260807 08:42:17.433977 29758 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:17.434026 29758 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:17.434077 29758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:17.435642 29758 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7c7f9f382441a4bca718496957a083" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42933 } }
I20260807 08:42:17.435791 29758 leader_election.cc:302] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa7c7f9f382441a4bca718496957a083; no voters:
I20260807 08:42:17.436003 29758 leader_election.cc:288] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:17.436173 29764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:17.436352 29758 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:17.436544 29764 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [term 1 LEADER]: Becoming Leader. State: Replica: aa7c7f9f382441a4bca718496957a083, State: Running, Role: LEADER
I20260807 08:42:17.436690 29764 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7c7f9f382441a4bca718496957a083" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42933 } }
I20260807 08:42:17.437079 29767 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa7c7f9f382441a4bca718496957a083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7c7f9f382441a4bca718496957a083" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42933 } } }
I20260807 08:42:17.437067 29764 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa7c7f9f382441a4bca718496957a083. Latest consensus state: current_term: 1 leader_uuid: "aa7c7f9f382441a4bca718496957a083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7c7f9f382441a4bca718496957a083" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42933 } } }
I20260807 08:42:17.437180 29767 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:17.437184 29764 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:17.437772 29774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:17.439162 29774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:17.441497 29774 catalog_manager.cc:1347] Generated new cluster ID: 31bd6281de8342869eb7a3a3f595cf05
I20260807 08:42:17.441560 29774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:42:17 dist-test-slave-9zdj krb5kdc[29672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:17.447572 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:17.431947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.28.108.190:42933)
0807 08:42:17.432158 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:17.432170 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:17.432219 (+ 49us) server_negotiation.cc:407] Connection header received
0807 08:42:17.432985 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:17.432997 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:17.433080 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:17.433273 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:17.434340 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:17.435910 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:17.437151 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:17.437887 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:17.438020 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:17.443183 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:17.443188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:17.443189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:17.443204 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:17.445485 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:17.445998 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:17.446003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:17.446003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:17.446163 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:17.446424 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:17.446425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:17.446426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:17.446819 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:17.447021 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:17.447126 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 08:42:17.447326 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
I20260807 08:42:17.463281 29774 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:17.464326 29774 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:17.474488 29774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa7c7f9f382441a4bca718496957a083: Generated new TSK 0
I20260807 08:42:17.474819 29774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:42:17.519598 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42933
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:36835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:17.638388 29788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:17.638736 29788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:17.638770 29788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:17.640177 29788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:17.640229 29788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:17.640261 29788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:17.641992 29788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:36835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:17.642235 29788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:17.642452 29788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:17.648280 29797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:17.648260 29799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:17.648260 29794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:17.648293 29788 server_base.cc:1034] running on GCE node
I20260807 08:42:17.648852 29788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:17.649139 29788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:17.650332 29788 hybrid_clock.cc:648] HybridClock initialized: now 1786092137650327 us; error 43 us; skew 500 ppm
Aug 07 08:42:17 dist-test-slave-9zdj krb5kdc[29672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:17.653316 29788 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:42:17.653906 29788 webserver.cc:460] Webserver started at http://127.28.108.129:44297/ using document root <none> and password file <none>
I20260807 08:42:17.654116 29788 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:17.654162 29788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:17.654263 29788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:17.655329 29788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "e04d8bd8e4874d198639a27031e50ad4"
format_stamp: "Formatted at 2026-08-07 08:42:17 on dist-test-slave-9zdj"
server_key: "5a04d9bfb33cd4194c4efddf7079b414"
server_key_iv: "20740cca05c509487921165e41fe2f8b"
server_key_version: "encryptionkey@0"
I20260807 08:42:17.655647 29788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "e04d8bd8e4874d198639a27031e50ad4"
format_stamp: "Formatted at 2026-08-07 08:42:17 on dist-test-slave-9zdj"
server_key: "5a04d9bfb33cd4194c4efddf7079b414"
server_key_iv: "20740cca05c509487921165e41fe2f8b"
server_key_version: "encryptionkey@0"
I20260807 08:42:17.657094 29788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:42:17.658038 29806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:17.658270 29788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:17.658372 29788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "e04d8bd8e4874d198639a27031e50ad4"
format_stamp: "Formatted at 2026-08-07 08:42:17 on dist-test-slave-9zdj"
server_key: "5a04d9bfb33cd4194c4efddf7079b414"
server_key_iv: "20740cca05c509487921165e41fe2f8b"
server_key_version: "encryptionkey@0"
I20260807 08:42:17.658479 29788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:17.667387 29788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:17.670783 29788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:17.670950 29788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:17.671221 29788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:17.671619 29788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:17.671669 29788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:17.671726 29788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:17.671756 29788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:17.677965 29788 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:36807
I20260807 08:42:17.678125 29926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:36807 every 8 connection(s)
I20260807 08:42:17.678439 29788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:42:17.679318 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 29788
I20260807 08:42:17.679445 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:17.679642 29106 external_mini_cluster.cc:1428] Setting key 702ef3959916fe336664d7f55a539e3e
Aug 07 08:42:17 dist-test-slave-9zdj krb5kdc[29672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
I20260807 08:42:17.689911 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:17.679438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:50667 (local address 127.28.108.190:42933)
0807 08:42:17.679499 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:17.679501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:17.679790 (+ 289us) server_negotiation.cc:407] Connection header received
0807 08:42:17.681115 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:17.681116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:17.681129 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:17.681178 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:17.682020 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:17.682644 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:17.683873 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:17.684131 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:17.684233 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:17.686793 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:17.686794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:17.686794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:17.686803 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:17.688309 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:17.688796 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:17.688798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:17.688798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:17.688863 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:17.689143 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:17.689144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:17.689145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:17.689276 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:17.689361 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:17.689706 (+ 345us) server_negotiation.cc:299] Negotiation successful
0807 08:42:17.689809 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 08:42:17.690140 29929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:17.679434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42933 (local address 127.28.108.129:50667)
0807 08:42:17.679630 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:17.679675 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:17.680994 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:17.681272 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:17.681275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:17.681608 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:17.681910 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:17.681915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:17.682773 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:17.682774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:17.683755 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:17.683758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:17.684251 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:17.684252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:17.684403 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:17.685008 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:17.685018 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:17.686681 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:17.688421 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:17.688430 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:17.688433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:17.688692 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:17.688942 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:17.688943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:17.688943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:17.689066 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:17.689364 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:17.689372 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:17.689530 (+ 158us) client_negotiation.cc:769] Sending connection context
0807 08:42:17.689681 (+ 151us) client_negotiation.cc:240] Negotiation successful
0807 08:42:17.689875 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260807 08:42:17.690903 29927 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42933
I20260807 08:42:17.691040 29927 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:17.691299 29927 heartbeater.cc:511] Master 127.28.108.190:42933 requested a full tablet report, sending...
I20260807 08:42:17.692039 29720 ts_manager.cc:194] Registered new tserver with Master: e04d8bd8e4874d198639a27031e50ad4 (127.28.108.129:36807)
I20260807 08:42:17.692960 29720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:50667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:42:17.714112 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42933
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:36835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:17.805951 29932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:17.806187 29932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:17.806216 29932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:17.807607 29932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:17.807698 29932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:17.807732 29932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:17.809425 29932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:36835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:17.809715 29932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:17.809933 29932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:42:17.815852 29932 server_base.cc:1034] running on GCE node
W20260807 08:42:17.815958 29939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:17.816082 29941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:17.815948 29938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:17.816359 29932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:17.816599 29932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:17.817828 29932 hybrid_clock.cc:648] HybridClock initialized: now 1786092137817791 us; error 52 us; skew 500 ppm
Aug 07 08:42:17 dist-test-slave-9zdj krb5kdc[29672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:17.821240 29932 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:42:17.821907 29932 webserver.cc:460] Webserver started at http://127.28.108.130:33903/ using document root <none> and password file <none>
I20260807 08:42:17.822168 29932 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:17.822252 29932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:17.822417 29932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:17.823594 29932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "382b1b0d3ea14c71aa214efd1de5e9f1"
format_stamp: "Formatted at 2026-08-07 08:42:17 on dist-test-slave-9zdj"
server_key: "a039c78ad321823a30092c0fd4155fda"
server_key_iv: "e433bab46608bc46bc2061d7bf47b0fd"
server_key_version: "encryptionkey@0"
I20260807 08:42:17.824010 29932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "382b1b0d3ea14c71aa214efd1de5e9f1"
format_stamp: "Formatted at 2026-08-07 08:42:17 on dist-test-slave-9zdj"
server_key: "a039c78ad321823a30092c0fd4155fda"
server_key_iv: "e433bab46608bc46bc2061d7bf47b0fd"
server_key_version: "encryptionkey@0"
I20260807 08:42:17.825914 29932 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 08:42:17.827052 29949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:17.827361 29932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:17.827512 29932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "382b1b0d3ea14c71aa214efd1de5e9f1"
format_stamp: "Formatted at 2026-08-07 08:42:17 on dist-test-slave-9zdj"
server_key: "a039c78ad321823a30092c0fd4155fda"
server_key_iv: "e433bab46608bc46bc2061d7bf47b0fd"
server_key_version: "encryptionkey@0"
I20260807 08:42:17.827608 29932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:17.844271 29932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:17.846858 29932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:17.847101 29932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:17.847388 29932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:17.847741 29932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:17.847806 29932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:17.847855 29932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:17.847895 29932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:17.854825 29932 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:41921
I20260807 08:42:17.854866 30074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:41921 every 8 connection(s)
I20260807 08:42:17.855255 29932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:17.861483 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 29932
I20260807 08:42:17.861577 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:42:17.861959 29106 external_mini_cluster.cc:1428] Setting key 8a13eda0f90ba8101a230625fe3f75f0
Aug 07 08:42:17 dist-test-slave-9zdj krb5kdc[29672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:17.868418 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:17.856105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:40753 (local address 127.28.108.190:42933)
0807 08:42:17.856191 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:17.856193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:17.856396 (+ 203us) server_negotiation.cc:407] Connection header received
0807 08:42:17.857473 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:17.857474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:17.857488 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:17.857562 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:17.858365 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:17.858981 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:17.860223 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:17.860449 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:17.860523 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:17.864078 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:17.864080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:17.864080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:17.864089 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:17.866212 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:17.866748 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:17.866750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:17.866750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:17.866823 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:17.867399 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:17.867401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:17.867401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:17.867609 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:17.867716 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:17.868095 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 08:42:17.868246 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 08:42:17.868462 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:17.856068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42933 (local address 127.28.108.130:40753)
0807 08:42:17.856240 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:17.856286 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:17.857327 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:17.857711 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:17.857716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:17.858023 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:17.858242 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:17.858247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:17.859087 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:17.859089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:17.860111 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:17.860115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:17.860602 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:17.860603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:17.860730 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:17.861910 (+ 1180us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:17.861924 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:17.863952 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:17.866334 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:17.866343 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:17.866346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:17.866636 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:17.866923 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:17.866924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:17.866924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:17.867119 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:17.867732 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:17.867740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:17.867913 (+ 173us) client_negotiation.cc:769] Sending connection context
0807 08:42:17.868019 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 08:42:17.868258 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260807 08:42:17.869082 30075 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42933
I20260807 08:42:17.869225 30075 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:17.869438 30075 heartbeater.cc:511] Master 127.28.108.190:42933 requested a full tablet report, sending...
I20260807 08:42:17.870009 29720 ts_manager.cc:194] Registered new tserver with Master: 382b1b0d3ea14c71aa214efd1de5e9f1 (127.28.108.130:41921)
I20260807 08:42:17.870517 29720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:40753
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:42:17.897305 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42933
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:36835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:17.991784 30081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:17.992025 30081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:17.992056 30081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:17.993496 30081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:17.993567 30081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:17.993600 30081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:17.995478 30081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:36835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:17.995792 30081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:17.996073 30081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:18.001834 30087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:18.002270 30090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:18.002328 30081 server_base.cc:1034] running on GCE node
W20260807 08:42:18.002240 30088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:18.002702 30081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:18.002972 30081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:18.004146 30081 hybrid_clock.cc:648] HybridClock initialized: now 1786092138004068 us; error 93 us; skew 500 ppm
Aug 07 08:42:18 dist-test-slave-9zdj krb5kdc[29672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092138, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:18.008023 30081 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:42:18.008742 30081 webserver.cc:460] Webserver started at http://127.28.108.131:42559/ using document root <none> and password file <none>
I20260807 08:42:18.009088 30081 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:18.009178 30081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:18.009397 30081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:18.010891 30081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "142da74beb734888a9593a12fd9e1aee"
format_stamp: "Formatted at 2026-08-07 08:42:18 on dist-test-slave-9zdj"
server_key: "2f31c343914162810ac092454fcbae53"
server_key_iv: "33b1331a20d1edd1d4e96fe7535f161f"
server_key_version: "encryptionkey@0"
I20260807 08:42:18.011343 30081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "142da74beb734888a9593a12fd9e1aee"
format_stamp: "Formatted at 2026-08-07 08:42:18 on dist-test-slave-9zdj"
server_key: "2f31c343914162810ac092454fcbae53"
server_key_iv: "33b1331a20d1edd1d4e96fe7535f161f"
server_key_version: "encryptionkey@0"
I20260807 08:42:18.013028 30081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:42:18.014168 30098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:18.014514 30081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:18.014653 30081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "142da74beb734888a9593a12fd9e1aee"
format_stamp: "Formatted at 2026-08-07 08:42:18 on dist-test-slave-9zdj"
server_key: "2f31c343914162810ac092454fcbae53"
server_key_iv: "33b1331a20d1edd1d4e96fe7535f161f"
server_key_version: "encryptionkey@0"
I20260807 08:42:18.014750 30081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:18.038168 30081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:18.040966 30081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:18.041194 30081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:18.041594 30081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:18.042055 30081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:18.042121 30081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:18.042160 30081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:18.042194 30081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:18.049489 30081 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:34163
I20260807 08:42:18.049525 30221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:34163 every 8 connection(s)
I20260807 08:42:18.050056 30081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:18.055181 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 30081
I20260807 08:42:18.055409 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:18.055613 29106 external_mini_cluster.cc:1428] Setting key 051be969bb6b48ab20eab86f65e18479
Aug 07 08:42:18 dist-test-slave-9zdj krb5kdc[29672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092138, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:18.061923 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.050982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:51355 (local address 127.28.108.190:42933)
0807 08:42:18.051054 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:18.051056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:18.051413 (+ 357us) server_negotiation.cc:407] Connection header received
0807 08:42:18.052435 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:18.052436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:18.052448 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:18.052522 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:18.053367 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.054060 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.055348 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.055624 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.055708 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.058401 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:18.058403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:18.058404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:18.058412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:18.060097 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:18.060714 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:18.060716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:18.060716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:18.060795 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:18.061077 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:18.061078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:18.061078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:18.061220 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:18.061301 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:18.061647 (+ 346us) server_negotiation.cc:299] Negotiation successful
0807 08:42:18.061743 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 08:42:18.062109 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.050949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42933 (local address 127.28.108.131:51355)
0807 08:42:18.051161 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:18.051211 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:18.052314 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:18.052618 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:18.052622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:18.053010 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:18.053245 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.053251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.054167 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.054168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.055221 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.055224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.055726 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.055728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.055915 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.056632 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:18.056643 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:18.058291 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:18.060327 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:18.060339 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:18.060341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:18.060621 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:18.060875 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:18.060876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:18.060876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:18.061000 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:18.061290 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:18.061298 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:18.061470 (+ 172us) client_negotiation.cc:769] Sending connection context
0807 08:42:18.061583 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 08:42:18.061829 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":120,"threads_started":1}
I20260807 08:42:18.063047 30222 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42933
I20260807 08:42:18.063194 30222 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:18.063416 30222 heartbeater.cc:511] Master 127.28.108.190:42933 requested a full tablet report, sending...
I20260807 08:42:18.063990 29720 ts_manager.cc:194] Registered new tserver with Master: 142da74beb734888a9593a12fd9e1aee (127.28.108.131:34163)
I20260807 08:42:18.064512 29720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:51355
I20260807 08:42:18.068176 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:42:18.080410 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.072034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.28.108.190:42933)
0807 08:42:18.072126 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:18.072128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:18.072207 (+ 79us) server_negotiation.cc:407] Connection header received
0807 08:42:18.072270 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:18.072271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:18.072283 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:18.072362 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:18.073001 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.073540 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.074623 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.074880 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.074953 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.076627 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:18.076629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:18.076629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:18.076637 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:18.078655 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:18.079269 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:18.079271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:18.079271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:18.079347 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:18.079626 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:18.079627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:18.079628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:18.079851 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:18.079951 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:18.080109 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 08:42:18.080326 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 08:42:18.083865 29720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:18.088460 29720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:18.097378 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.092648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.28.108.130:41921)
0807 08:42:18.093099 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:18.093103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:18.093112 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:42:18.093150 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:18.093153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:18.093169 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:18.093530 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:18.093654 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.094540 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.096299 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.096834 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.096933 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.097005 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:18.097226 (+ 221us) server_negotiation.cc:299] Negotiation successful
0807 08:42:18.097278 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":96,"threads_started":1}
I20260807 08:42:18.097386 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.092487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41921 (local address 127.0.0.1:35104)
0807 08:42:18.092898 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:18.092909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:18.092962 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:18.093270 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:18.093271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:18.093273 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:18.093444 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.093446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.094661 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.094662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.096206 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.096209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.096933 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.096934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.097033 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.097049 (+ 16us) client_negotiation.cc:769] Sending connection context
0807 08:42:18.097125 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 08:42:18.097196 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":111,"thread_start_us":110,"threads_started":1}
I20260807 08:42:18.097404 30233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.092734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:34163 (local address 127.0.0.1:60130)
0807 08:42:18.092999 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:18.093010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:18.093063 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:18.093372 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:18.093373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:18.093375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:18.093420 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.093421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.094796 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.094797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.096136 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.096139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.097105 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.097106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.097164 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.097168 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:18.097218 (+ 50us) client_negotiation.cc:240] Negotiation successful
0807 08:42:18.097273 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":55,"thread_start_us":77,"threads_started":1}
I20260807 08:42:18.097535 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.092737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.28.108.129:36807)
0807 08:42:18.093041 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:18.093048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:18.093060 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:42:18.093107 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:18.093109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:18.093129 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:18.093221 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:18.093449 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.094674 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.096064 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.096887 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.096969 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.097034 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:18.097384 (+ 350us) server_negotiation.cc:299] Negotiation successful
0807 08:42:18.097440 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
I20260807 08:42:18.097699 30234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.092478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:36807 (local address 127.0.0.1:35614)
0807 08:42:18.092679 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:18.092735 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:18.092818 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:18.093233 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:18.093237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:18.093239 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:18.093337 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.093341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.094828 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.094830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.095988 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.095992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.097074 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.097075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.097155 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.097162 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 08:42:18.097212 (+ 50us) client_negotiation.cc:240] Negotiation successful
0807 08:42:18.097263 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":108,"thread_start_us":86,"threads_started":1}
I20260807 08:42:18.098613 29856 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab149fc6e1b4775b30c891cfef13d03 (DEFAULT_TABLE table=test-table [id=e96ef270d12d4712b6d9e001dea8a8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:18.098588 30002 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab149fc6e1b4775b30c891cfef13d03 (DEFAULT_TABLE table=test-table [id=e96ef270d12d4712b6d9e001dea8a8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:18.099014 29856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ab149fc6e1b4775b30c891cfef13d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:18.099025 30002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ab149fc6e1b4775b30c891cfef13d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:18.101222 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.092981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.28.108.131:34163)
0807 08:42:18.093187 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:18.093193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:18.093203 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:18.093240 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:18.093242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:18.093258 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:18.093370 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:18.093500 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.094669 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.096250 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.096990 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.097095 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.097154 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:18.097939 (+ 785us) server_negotiation.cc:299] Negotiation successful
0807 08:42:18.098002 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":110,"threads_started":1}
I20260807 08:42:18.102180 30240 tablet_bootstrap.cc:492] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4: Bootstrap starting.
I20260807 08:42:18.102123 30148 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab149fc6e1b4775b30c891cfef13d03 (DEFAULT_TABLE table=test-table [id=e96ef270d12d4712b6d9e001dea8a8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:18.102607 30148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ab149fc6e1b4775b30c891cfef13d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:18.102654 30241 tablet_bootstrap.cc:492] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1: Bootstrap starting.
I20260807 08:42:18.102914 30240 tablet_bootstrap.cc:654] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:18.103340 30241 tablet_bootstrap.cc:654] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:18.103597 30240 log.cc:824] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:18.103992 30241 log.cc:824] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:18.104938 30241 tablet_bootstrap.cc:492] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1: No bootstrap required, opened a new log
I20260807 08:42:18.105032 30241 ts_tablet_manager.cc:1397] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:42:18.106212 30240 tablet_bootstrap.cc:492] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4: No bootstrap required, opened a new log
I20260807 08:42:18.106316 30240 ts_tablet_manager.cc:1397] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 08:42:18.106705 30244 tablet_bootstrap.cc:492] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee: Bootstrap starting.
I20260807 08:42:18.107509 30244 tablet_bootstrap.cc:654] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:18.107443 30241 raft_consensus.cc:348] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41921 } } peers { permanent_uuid: "e04d8bd8e4874d198639a27031e50ad4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 36807 } } peers { permanent_uuid: "142da74beb734888a9593a12fd9e1aee" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34163 } }
I20260807 08:42:18.107686 30241 raft_consensus.cc:374] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:18.107734 30241 raft_consensus.cc:729] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382b1b0d3ea14c71aa214efd1de5e9f1, State: Initialized, Role: FOLLOWER
I20260807 08:42:18.107861 30241 consensus_queue.cc:260] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41921 } } peers { permanent_uuid: "e04d8bd8e4874d198639a27031e50ad4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 36807 } } peers { permanent_uuid: "142da74beb734888a9593a12fd9e1aee" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34163 } }
I20260807 08:42:18.108115 30244 log.cc:824] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:18.108314 30075 heartbeater.cc:503] Master 127.28.108.190:42933 was elected leader, sending a full tablet report...
I20260807 08:42:18.108240 30241 ts_tablet_manager.cc:1428] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 08:42:18.108868 30240 raft_consensus.cc:348] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41921 } } peers { permanent_uuid: "e04d8bd8e4874d198639a27031e50ad4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 36807 } } peers { permanent_uuid: "142da74beb734888a9593a12fd9e1aee" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34163 } }
I20260807 08:42:18.109012 30240 raft_consensus.cc:374] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:18.109033 30240 raft_consensus.cc:729] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04d8bd8e4874d198639a27031e50ad4, State: Initialized, Role: FOLLOWER
I20260807 08:42:18.109097 30244 tablet_bootstrap.cc:492] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee: No bootstrap required, opened a new log
I20260807 08:42:18.109166 30244 ts_tablet_manager.cc:1397] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:42:18.109197 30240 consensus_queue.cc:260] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41921 } } peers { permanent_uuid: "e04d8bd8e4874d198639a27031e50ad4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 36807 } } peers { permanent_uuid: "142da74beb734888a9593a12fd9e1aee" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34163 } }
I20260807 08:42:18.109508 30240 ts_tablet_manager.cc:1428] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 08:42:18.109513 29927 heartbeater.cc:503] Master 127.28.108.190:42933 was elected leader, sending a full tablet report...
I20260807 08:42:18.111285 30244 raft_consensus.cc:348] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41921 } } peers { permanent_uuid: "e04d8bd8e4874d198639a27031e50ad4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 36807 } } peers { permanent_uuid: "142da74beb734888a9593a12fd9e1aee" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34163 } }
I20260807 08:42:18.111435 30244 raft_consensus.cc:374] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:18.111457 30244 raft_consensus.cc:729] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142da74beb734888a9593a12fd9e1aee, State: Initialized, Role: FOLLOWER
I20260807 08:42:18.111554 30244 consensus_queue.cc:260] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41921 } } peers { permanent_uuid: "e04d8bd8e4874d198639a27031e50ad4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 36807 } } peers { permanent_uuid: "142da74beb734888a9593a12fd9e1aee" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34163 } }
I20260807 08:42:18.111788 30244 ts_tablet_manager.cc:1428] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:42:18.111845 30222 heartbeater.cc:503] Master 127.28.108.190:42933 was elected leader, sending a full tablet report...
W20260807 08:42:18.179313 29928 tablet.cc:2369] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:42:18.300922 30223 tablet.cc:2369] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:18.347078 30245 raft_consensus.cc:482] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:18.347244 30245 raft_consensus.cc:504] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41921 } } peers { permanent_uuid: "e04d8bd8e4874d198639a27031e50ad4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 36807 } } peers { permanent_uuid: "142da74beb734888a9593a12fd9e1aee" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34163 } }
I20260807 08:42:18.347988 30245 leader_election.cc:288] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e04d8bd8e4874d198639a27031e50ad4 (127.28.108.129:36807), 142da74beb734888a9593a12fd9e1aee (127.28.108.131:34163)
I20260807 08:42:18.351503 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.348468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:49133 (local address 127.28.108.129:36807)
0807 08:42:18.348529 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:18.348532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:18.348545 (+ 13us) server_negotiation.cc:407] Connection header received
0807 08:42:18.348582 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:18.348584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:18.348602 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:18.348708 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:18.348959 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.349521 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.350652 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.351070 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.351174 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.351185 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:18.351374 (+ 189us) server_negotiation.cc:299] Negotiation successful
0807 08:42:18.351418 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 08:42:18.351540 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.348253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:36807 (local address 127.28.108.130:49133)
0807 08:42:18.348401 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:18.348415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:18.348469 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:18.348735 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:18.348737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:18.348740 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:18.348825 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.348827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.349653 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.349654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.350530 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.350533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.351178 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.351179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.351272 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.351275 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:18.351356 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 08:42:18.351406 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260807 08:42:18.351567 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.348416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:34163 (local address 127.28.108.130:49283)
0807 08:42:18.348559 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:18.348569 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:18.348623 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:18.348902 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:18.348904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:18.348906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:18.348999 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.349001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.349867 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.349868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.350636 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:18.350638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.351262 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.351263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:18.351346 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.351349 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:18.351416 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 08:42:18.351459 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260807 08:42:18.351547 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.348546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:49283 (local address 127.28.108.131:34163)
0807 08:42:18.348620 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:18.348622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:18.348633 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:42:18.348685 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:18.348687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:18.348697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:18.348896 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:18.349078 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.349765 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.350746 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.351157 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.351261 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.351271 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:18.351436 (+ 165us) server_negotiation.cc:299] Negotiation successful
0807 08:42:18.351476 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 08:42:18.352030 30175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab149fc6e1b4775b30c891cfef13d03" candidate_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142da74beb734888a9593a12fd9e1aee" is_pre_election: true
I20260807 08:42:18.352262 30175 raft_consensus.cc:2454] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 382b1b0d3ea14c71aa214efd1de5e9f1 in term 0.
I20260807 08:42:18.352576 29950 leader_election.cc:302] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142da74beb734888a9593a12fd9e1aee, 382b1b0d3ea14c71aa214efd1de5e9f1; no voters:
I20260807 08:42:18.352748 30245 raft_consensus.cc:2781] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:18.352851 30245 raft_consensus.cc:482] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:18.352885 30245 raft_consensus.cc:3037] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:18.353734 30245 raft_consensus.cc:504] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41921 } } peers { permanent_uuid: "e04d8bd8e4874d198639a27031e50ad4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 36807 } } peers { permanent_uuid: "142da74beb734888a9593a12fd9e1aee" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34163 } }
I20260807 08:42:18.353909 30245 leader_election.cc:288] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [CANDIDATE]: Term 1 election: Requested vote from peers e04d8bd8e4874d198639a27031e50ad4 (127.28.108.129:36807), 142da74beb734888a9593a12fd9e1aee (127.28.108.131:34163)
I20260807 08:42:18.352029 29878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab149fc6e1b4775b30c891cfef13d03" candidate_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04d8bd8e4874d198639a27031e50ad4" is_pre_election: true
I20260807 08:42:18.354079 29878 raft_consensus.cc:2454] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 382b1b0d3ea14c71aa214efd1de5e9f1 in term 0.
I20260807 08:42:18.354184 30175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab149fc6e1b4775b30c891cfef13d03" candidate_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142da74beb734888a9593a12fd9e1aee"
I20260807 08:42:18.354183 29877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab149fc6e1b4775b30c891cfef13d03" candidate_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04d8bd8e4874d198639a27031e50ad4"
I20260807 08:42:18.354287 29877 raft_consensus.cc:3037] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:18.354290 30175 raft_consensus.cc:3037] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:18.355074 29877 raft_consensus.cc:2454] T 4ab149fc6e1b4775b30c891cfef13d03 P e04d8bd8e4874d198639a27031e50ad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 382b1b0d3ea14c71aa214efd1de5e9f1 in term 1.
I20260807 08:42:18.355242 30175 raft_consensus.cc:2454] T 4ab149fc6e1b4775b30c891cfef13d03 P 142da74beb734888a9593a12fd9e1aee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 382b1b0d3ea14c71aa214efd1de5e9f1 in term 1.
I20260807 08:42:18.355252 29950 leader_election.cc:302] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382b1b0d3ea14c71aa214efd1de5e9f1, e04d8bd8e4874d198639a27031e50ad4; no voters:
I20260807 08:42:18.355406 30245 raft_consensus.cc:2781] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:18.355598 30245 raft_consensus.cc:686] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [term 1 LEADER]: Becoming Leader. State: Replica: 382b1b0d3ea14c71aa214efd1de5e9f1, State: Running, Role: LEADER
I20260807 08:42:18.355706 30245 consensus_queue.cc:237] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41921 } } peers { permanent_uuid: "e04d8bd8e4874d198639a27031e50ad4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 36807 } } peers { permanent_uuid: "142da74beb734888a9593a12fd9e1aee" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34163 } }
W20260807 08:42:18.356168 30076 tablet.cc:2369] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:18.356824 29720 catalog_manager.cc:5495] T 4ab149fc6e1b4775b30c891cfef13d03 P 382b1b0d3ea14c71aa214efd1de5e9f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 382b1b0d3ea14c71aa214efd1de5e9f1 (127.28.108.130). New cstate: current_term: 1 leader_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382b1b0d3ea14c71aa214efd1de5e9f1" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e04d8bd8e4874d198639a27031e50ad4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 36807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142da74beb734888a9593a12fd9e1aee" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34163 } health_report { overall_health: UNKNOWN } } }
Aug 07 08:42:18 dist-test-slave-9zdj krb5kdc[29672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
I20260807 08:42:18.395300 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.386880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.28.108.129:36807)
0807 08:42:18.386986 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:18.386988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:18.387049 (+ 61us) server_negotiation.cc:407] Connection header received
0807 08:42:18.387100 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:18.387101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:18.387111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:18.387165 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:18.387846 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.388342 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.389125 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.389348 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.389416 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.392151 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:18.392168 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:18.392168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:18.392175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:18.393973 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:18.394356 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:18.394359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:18.394359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:18.394426 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:18.394673 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:18.394674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:18.394675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:18.394869 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:18.395012 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:18.395083 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 08:42:18.395163 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Aug 07 08:42:18 dist-test-slave-9zdj krb5kdc[29672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092138, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 08:42:18 dist-test-slave-9zdj krb5kdc[29672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092138, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
I20260807 08:42:18.426723 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.417202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.28.108.129:36807)
0807 08:42:18.417358 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:18.417360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:18.417375 (+ 15us) server_negotiation.cc:407] Connection header received
0807 08:42:18.417497 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:18.417498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:18.417511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:18.417591 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:18.418437 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.419012 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.419920 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.420171 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.420275 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.423232 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:18.423234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:18.423234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:18.423245 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:18.424950 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:18.425596 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:18.425598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:18.425599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:18.425728 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:18.425983 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:18.425984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:18.425985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:18.426132 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:18.426285 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:18.426505 (+ 220us) server_negotiation.cc:299] Negotiation successful
0807 08:42:18.426620 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
Aug 07 08:42:18 dist-test-slave-9zdj krb5kdc[29672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 08:42:18 dist-test-slave-9zdj krb5kdc[29672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
I20260807 08:42:18.458564 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:18.449649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.28.108.129:36807)
0807 08:42:18.449728 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:18.449730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:18.449911 (+ 181us) server_negotiation.cc:407] Connection header received
0807 08:42:18.449949 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:18.449950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:18.449962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:18.450031 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:18.450828 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.451391 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.452231 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:18.452487 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:18.452549 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:18.455321 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:18.455324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:18.455324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:18.455335 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:18.457125 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:18.457699 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:18.457701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:18.457701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:18.457774 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:18.458052 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:18.458053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:18.458053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:18.458185 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:18.458280 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:18.458405 (+ 125us) server_negotiation.cc:299] Negotiation successful
0807 08:42:18.458480 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260807 08:42:18.459467 29835 tablet_service.cc:2908] Rejecting scan request for tablet 4ab149fc6e1b4775b30c891cfef13d03: Uninitialized: clean time has not yet been initialized
I20260807 08:42:18.462158 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 29788
I20260807 08:42:18.470877 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 29932
I20260807 08:42:18.478879 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 30081
I20260807 08:42:18.486840 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 29686
2026-08-07T08:42:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2493 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:18 dist-test-slave-9zdj krb5kdc[30265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:18 dist-test-slave-9zdj krb5kdc[30265](info): set up 2 sockets
Aug 07 08:42:18 dist-test-slave-9zdj krb5kdc[30265](info): commencing operation
krb5kdc: starting...
W20260807 08:42:19.655808 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:19 dist-test-slave-9zdj krb5kdc[30265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:19.810568 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34059
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:34059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:19.908129 30278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:19.908445 30278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:19.908495 30278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:19.909930 30278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:19.910028 30278 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 08:42:19.910053 30278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:19.910077 30278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:19.910115 30278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:19.910153 30278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:19.912016 30278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:34059
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:19.912392 30278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:19.912686 30278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:19.919440 30283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:19.919423 30286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:19.919508 30284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:19.919638 30278 server_base.cc:1034] running on GCE node
I20260807 08:42:19.920086 30278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:19.920440 30278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:19.921705 30278 hybrid_clock.cc:648] HybridClock initialized: now 1786092139921685 us; error 34 us; skew 500 ppm
Aug 07 08:42:19 dist-test-slave-9zdj krb5kdc[30265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:19.933753 30278 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:19.934420 30278 webserver.cc:460] Webserver started at http://127.28.108.190:46387/ using document root <none> and password file <none>
I20260807 08:42:19.934700 30278 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:19.934774 30278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:19.934948 30278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:19.936127 30278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "f2230b2d94d842608bd4fe624b1dc24e"
format_stamp: "Formatted at 2026-08-07 08:42:19 on dist-test-slave-9zdj"
server_key: "876d89b0583a35323330b3a3f10a2dc9"
server_key_iv: "9cb17b94e84d3c2019c762914c9b617c"
server_key_version: "encryptionkey@0"
I20260807 08:42:19.936532 30278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "f2230b2d94d842608bd4fe624b1dc24e"
format_stamp: "Formatted at 2026-08-07 08:42:19 on dist-test-slave-9zdj"
server_key: "876d89b0583a35323330b3a3f10a2dc9"
server_key_iv: "9cb17b94e84d3c2019c762914c9b617c"
server_key_version: "encryptionkey@0"
I20260807 08:42:19.938066 30278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260807 08:42:19.938933 30294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:19.939159 30278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:19.939278 30278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "f2230b2d94d842608bd4fe624b1dc24e"
format_stamp: "Formatted at 2026-08-07 08:42:19 on dist-test-slave-9zdj"
server_key: "876d89b0583a35323330b3a3f10a2dc9"
server_key_iv: "9cb17b94e84d3c2019c762914c9b617c"
server_key_version: "encryptionkey@0"
I20260807 08:42:19.939360 30278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:19.956863 30278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:19.959583 30278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:19.959774 30278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:19.965888 30278 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:34059
I20260807 08:42:19.965982 30363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:34059 every 8 connection(s)
I20260807 08:42:19.966420 30278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:19.967383 30364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:19.968117 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 30278
I20260807 08:42:19.968232 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:19.968400 29106 external_mini_cluster.cc:1428] Setting key ad47a39a72101f18191a9989db2007e3
I20260807 08:42:19.971028 30364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e: Bootstrap starting.
I20260807 08:42:19.971738 30364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:19.972072 30364 log.cc:824] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:19.972990 30364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e: No bootstrap required, opened a new log
I20260807 08:42:19.974802 30364 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2230b2d94d842608bd4fe624b1dc24e" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34059 } }
I20260807 08:42:19.975003 30364 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:19.975044 30364 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2230b2d94d842608bd4fe624b1dc24e, State: Initialized, Role: FOLLOWER
I20260807 08:42:19.975237 30364 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2230b2d94d842608bd4fe624b1dc24e" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34059 } }
I20260807 08:42:19.975347 30364 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:19.975389 30364 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:19.975437 30364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:19.976836 30364 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2230b2d94d842608bd4fe624b1dc24e" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34059 } }
I20260807 08:42:19.976977 30364 leader_election.cc:302] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2230b2d94d842608bd4fe624b1dc24e; no voters:
I20260807 08:42:19.977205 30364 leader_election.cc:288] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:19.977283 30372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:19.977474 30372 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [term 1 LEADER]: Becoming Leader. State: Replica: f2230b2d94d842608bd4fe624b1dc24e, State: Running, Role: LEADER
I20260807 08:42:19.977540 30364 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:19.977710 30372 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2230b2d94d842608bd4fe624b1dc24e" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34059 } }
Aug 07 08:42:19 dist-test-slave-9zdj krb5kdc[30265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:19.978142 30374 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2230b2d94d842608bd4fe624b1dc24e. Latest consensus state: current_term: 1 leader_uuid: "f2230b2d94d842608bd4fe624b1dc24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2230b2d94d842608bd4fe624b1dc24e" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34059 } } }
I20260807 08:42:19.978255 30374 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:19.978662 30379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:19.979480 30379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:19.982021 30379 catalog_manager.cc:1347] Generated new cluster ID: d1c3082833fa4f1e8d56b9d75c0ed2a2
I20260807 08:42:19.982110 30379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:19.983152 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:19.969047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.28.108.190:34059)
0807 08:42:19.969261 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:19.969275 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:19.969311 (+ 36us) server_negotiation.cc:407] Connection header received
0807 08:42:19.969846 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:19.969855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:19.969915 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:19.970066 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:19.971015 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:19.972618 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:19.973785 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:19.974910 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:19.975169 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:19.978531 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:19.978537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:19.978537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:19.978552 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:19.980914 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:19.981466 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:19.981470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:19.981470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:19.981591 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:19.981957 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:19.981958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:19.981958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:19.982366 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:19.982520 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:19.982684 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 08:42:19.982838 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":110,"threads_started":1}
I20260807 08:42:19.979666 30392 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 08:42:19.979144 30373 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2230b2d94d842608bd4fe624b1dc24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2230b2d94d842608bd4fe624b1dc24e" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34059 } } }
I20260807 08:42:19.985893 30373 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:20.004532 30379 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:20.005236 30379 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:20.014981 30379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e: Generated new TSK 0
I20260807 08:42:20.015239 30379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:42:20.057731 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34059
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:20.150842 30396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:20.151121 30396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:20.151154 30396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:20.152559 30396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:20.152626 30396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:20.152647 30396 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 08:42:20.152667 30396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:20.154402 30396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.108.190:34059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:20.154629 30396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:20.154841 30396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:20.161113 30405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:20.161092 30396 server_base.cc:1034] running on GCE node
W20260807 08:42:20.161062 30402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:20.161087 30403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:20.161814 30396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:20.162281 30396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:20.163518 30396 hybrid_clock.cc:648] HybridClock initialized: now 1786092140163539 us; error 71 us; skew 500 ppm
Aug 07 08:42:20 dist-test-slave-9zdj krb5kdc[30265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092140, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:20.166883 30396 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:42:20.167474 30396 webserver.cc:460] Webserver started at http://127.28.108.129:40839/ using document root <none> and password file <none>
I20260807 08:42:20.167740 30396 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:20.167851 30396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:20.168015 30396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:20.169183 30396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "0d51b176ade9445cb3cab8c7941aa2d4"
format_stamp: "Formatted at 2026-08-07 08:42:20 on dist-test-slave-9zdj"
server_key: "4ab9bff19c4b172977e7f8b0b1367ae3"
server_key_iv: "62aae4f0c083cc53c6c5d6a9508a48f0"
server_key_version: "encryptionkey@0"
I20260807 08:42:20.169772 30396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "0d51b176ade9445cb3cab8c7941aa2d4"
format_stamp: "Formatted at 2026-08-07 08:42:20 on dist-test-slave-9zdj"
server_key: "4ab9bff19c4b172977e7f8b0b1367ae3"
server_key_iv: "62aae4f0c083cc53c6c5d6a9508a48f0"
server_key_version: "encryptionkey@0"
I20260807 08:42:20.171434 30396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:20.172634 30414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:20.173046 30396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:20.173169 30396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "0d51b176ade9445cb3cab8c7941aa2d4"
format_stamp: "Formatted at 2026-08-07 08:42:20 on dist-test-slave-9zdj"
server_key: "4ab9bff19c4b172977e7f8b0b1367ae3"
server_key_iv: "62aae4f0c083cc53c6c5d6a9508a48f0"
server_key_version: "encryptionkey@0"
I20260807 08:42:20.173274 30396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:20.184291 30396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:20.186951 30396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:20.187158 30396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:20.187853 30396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:20.187920 30396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:20.187973 30396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:20.188016 30396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:20.196045 30396 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:39009
I20260807 08:42:20.196136 30538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:39009 every 8 connection(s)
Aug 07 08:42:20 dist-test-slave-9zdj krb5kdc[30265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092140, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:20.196588 30396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:42:20.200812 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.188585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:38301 (local address 127.28.108.190:34059)
0807 08:42:20.188667 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.188669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.188973 (+ 304us) server_negotiation.cc:407] Connection header received
0807 08:42:20.189682 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.189684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.189699 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.189761 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:20.190566 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.191465 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.192633 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.193024 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.193111 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.197017 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:20.197019 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:20.197020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:20.197030 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:20.198756 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.199226 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.199228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.199229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.199352 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.199836 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.199837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.199838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.200000 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:20.200094 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.200547 (+ 453us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.200701 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 08:42:20.201022 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.188594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34059 (local address 127.28.108.129:38301)
0807 08:42:20.188783 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:20.188827 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:20.189495 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:20.189876 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:20.189880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:20.190226 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:20.190450 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.190456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.191600 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.191601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.192529 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.192532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.193145 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.193146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.193323 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.194742 (+ 1419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:20.194755 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:20.196823 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:20.198865 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.198873 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.198876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.199136 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.199515 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.199517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.199517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.199742 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.200083 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:20.200092 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:20.200364 (+ 272us) client_negotiation.cc:769] Sending connection context
0807 08:42:20.200512 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 08:42:20.200729 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260807 08:42:20.201689 30539 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34059
I20260807 08:42:20.201818 30539 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:20.202029 30539 heartbeater.cc:511] Master 127.28.108.190:34059 requested a full tablet report, sending...
I20260807 08:42:20.202685 30312 ts_manager.cc:194] Registered new tserver with Master: 0d51b176ade9445cb3cab8c7941aa2d4 (127.28.108.129:39009)
I20260807 08:42:20.203588 30312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:38301
I20260807 08:42:20.205017 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 30396
I20260807 08:42:20.205113 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:20.205329 29106 external_mini_cluster.cc:1428] Setting key 609395dbb6613d035dcdd29a9b1c50c9
I20260807 08:42:20.210140 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.203117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34059 (local address 127.28.108.129:47859)
0807 08:42:20.203284 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:20.203296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:20.203365 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:20.203496 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:20.203497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:20.203729 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:20.203803 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.203806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.204618 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.204619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.205344 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.205347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.205895 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.205896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.205982 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.206555 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:20.206561 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:20.206934 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:20.208901 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.208902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.208903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.209106 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.209338 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.209339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.209340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.209411 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.209740 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:20.209742 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:20.209798 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 08:42:20.209902 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 08:42:20.210033 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260807 08:42:20.210139 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.203224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:47859 (local address 127.28.108.190:34059)
0807 08:42:20.203282 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.203285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.203362 (+ 77us) server_negotiation.cc:407] Connection header received
0807 08:42:20.203415 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.203416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.203426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.203494 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:20.203897 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.204497 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.205484 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.205797 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.205893 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.207063 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:20.207064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:20.207064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:20.207073 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:20.208777 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.209189 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.209190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.209191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.209259 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.209485 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.209486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.209487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.209675 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:20.209738 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.209934 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.210031 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:42:20.237205 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34059
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:20.336165 30549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:20.336437 30549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:20.336470 30549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:20.338065 30549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:20.338135 30549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:20.338168 30549 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 08:42:20.338191 30549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:20.340054 30549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.108.190:34059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:20.340394 30549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:20.340695 30549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:20.347532 30555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:20.347536 30558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:20.347532 30554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:20.347553 30549 server_base.cc:1034] running on GCE node
I20260807 08:42:20.348186 30549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:20.348460 30549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:20.349761 30549 hybrid_clock.cc:648] HybridClock initialized: now 1786092140349749 us; error 37 us; skew 500 ppm
Aug 07 08:42:20 dist-test-slave-9zdj krb5kdc[30265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092140, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:20.353133 30549 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:42:20.353916 30549 webserver.cc:460] Webserver started at http://127.28.108.130:41889/ using document root <none> and password file <none>
I20260807 08:42:20.354228 30549 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:20.354282 30549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:20.354440 30549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:20.355705 30549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "28a5d10a37e543f89606dd49e74c3c4b"
format_stamp: "Formatted at 2026-08-07 08:42:20 on dist-test-slave-9zdj"
server_key: "ae097c2bc79e03ec6bcf237ca1cf75ed"
server_key_iv: "50ad45d53f883a60425f55ab4c54ab84"
server_key_version: "encryptionkey@0"
I20260807 08:42:20.356137 30549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "28a5d10a37e543f89606dd49e74c3c4b"
format_stamp: "Formatted at 2026-08-07 08:42:20 on dist-test-slave-9zdj"
server_key: "ae097c2bc79e03ec6bcf237ca1cf75ed"
server_key_iv: "50ad45d53f883a60425f55ab4c54ab84"
server_key_version: "encryptionkey@0"
I20260807 08:42:20.357800 30549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 08:42:20.358747 30565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:20.359017 30549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:20.359117 30549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "28a5d10a37e543f89606dd49e74c3c4b"
format_stamp: "Formatted at 2026-08-07 08:42:20 on dist-test-slave-9zdj"
server_key: "ae097c2bc79e03ec6bcf237ca1cf75ed"
server_key_iv: "50ad45d53f883a60425f55ab4c54ab84"
server_key_version: "encryptionkey@0"
I20260807 08:42:20.359201 30549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:20.371410 30549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:20.374423 30549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:20.374604 30549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:20.375283 30549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:20.375330 30549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:20.375357 30549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:20.375372 30549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:20.383955 30549 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:42223
I20260807 08:42:20.384088 30692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:42223 every 8 connection(s)
I20260807 08:42:20.384514 30549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
Aug 07 08:42:20 dist-test-slave-9zdj krb5kdc[30265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092140, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:20.388700 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.376126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:40639 (local address 127.28.108.190:34059)
0807 08:42:20.376211 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.376214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.376469 (+ 255us) server_negotiation.cc:407] Connection header received
0807 08:42:20.377303 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.377305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.377320 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.377384 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:20.379349 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.379957 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.381939 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.382170 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.382277 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.385520 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:20.385522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:20.385522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:20.385531 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:20.387039 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.387513 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.387515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.387515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.387590 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.387891 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.387892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.387893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.388036 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:20.388168 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.388494 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.388593 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 08:42:20.388940 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.376110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34059 (local address 127.28.108.130:40639)
0807 08:42:20.376310 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:20.376358 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:20.377114 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:20.378228 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:20.378234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:20.378849 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:20.379202 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.379236 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.380090 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.380091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.381825 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.381829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.382293 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.382294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.382470 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.383239 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:20.383252 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:20.385394 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:20.387141 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.387150 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.387152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.387421 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.387672 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.387673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.387673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.387799 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.388159 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:20.388167 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:20.388334 (+ 167us) client_negotiation.cc:769] Sending connection context
0807 08:42:20.388471 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 08:42:20.388661 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":123,"threads_started":1}
I20260807 08:42:20.389878 30693 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34059
I20260807 08:42:20.390017 30693 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:20.390324 30693 heartbeater.cc:511] Master 127.28.108.190:34059 requested a full tablet report, sending...
I20260807 08:42:20.390815 30312 ts_manager.cc:194] Registered new tserver with Master: 28a5d10a37e543f89606dd49e74c3c4b (127.28.108.130:42223)
I20260807 08:42:20.391366 30312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:40639
I20260807 08:42:20.394654 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 30549
I20260807 08:42:20.394744 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:42:20.395005 29106 external_mini_cluster.cc:1428] Setting key 84235601edb429c641e509568be55fc7
I20260807 08:42:20.398617 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.391168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:49739 (local address 127.28.108.190:34059)
0807 08:42:20.391218 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.391221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.391321 (+ 100us) server_negotiation.cc:407] Connection header received
0807 08:42:20.391471 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.391472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.391484 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.391547 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:20.391992 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.392596 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.393502 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.393787 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.393858 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.395127 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:20.395129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:20.395129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:20.395136 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:20.397159 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.397675 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.397676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.397676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.397765 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.397982 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.397983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.397984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.398165 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:20.398246 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.398426 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.398512 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 08:42:20.398631 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.391087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34059 (local address 127.28.108.130:49739)
0807 08:42:20.391243 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:20.391255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:20.391328 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:20.391560 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:20.391562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:20.391821 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:20.391897 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.391900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.392806 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.392807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.393407 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.393409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.393880 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.393880 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.393947 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.394557 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:20.394570 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:20.395005 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:20.397365 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.397366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.397366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.397563 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.397840 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.397841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.397841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.397912 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.398257 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:20.398258 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:20.398316 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 08:42:20.398387 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 08:42:20.398515 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:42:20.431171 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34059
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:20.528656 30702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:20.528973 30702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:20.529026 30702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:20.530665 30702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:20.530781 30702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:20.530818 30702 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 08:42:20.530862 30702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:20.532714 30702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.108.190:34059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:20.533056 30702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:20.533345 30702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:20.539366 30707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:20.539425 30710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:20.539386 30708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:20.539479 30702 server_base.cc:1034] running on GCE node
I20260807 08:42:20.539858 30702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:20.540149 30702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:20.541370 30702 hybrid_clock.cc:648] HybridClock initialized: now 1786092140541342 us; error 46 us; skew 500 ppm
Aug 07 08:42:20 dist-test-slave-9zdj krb5kdc[30265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092140, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:20.544772 30702 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:42:20.545374 30702 webserver.cc:460] Webserver started at http://127.28.108.131:41911/ using document root <none> and password file <none>
I20260807 08:42:20.545693 30702 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:20.545779 30702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:20.545934 30702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:20.547087 30702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "e1a47b3bef2545c380091109888f73ca"
format_stamp: "Formatted at 2026-08-07 08:42:20 on dist-test-slave-9zdj"
server_key: "b7879008d0a66c6b26c16f8f525e6421"
server_key_iv: "8a44588628402f6141c44bb23a70f0e2"
server_key_version: "encryptionkey@0"
I20260807 08:42:20.547521 30702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "e1a47b3bef2545c380091109888f73ca"
format_stamp: "Formatted at 2026-08-07 08:42:20 on dist-test-slave-9zdj"
server_key: "b7879008d0a66c6b26c16f8f525e6421"
server_key_iv: "8a44588628402f6141c44bb23a70f0e2"
server_key_version: "encryptionkey@0"
I20260807 08:42:20.549105 30702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 08:42:20.550381 30717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:20.550657 30702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:42:20.550784 30702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "e1a47b3bef2545c380091109888f73ca"
format_stamp: "Formatted at 2026-08-07 08:42:20 on dist-test-slave-9zdj"
server_key: "b7879008d0a66c6b26c16f8f525e6421"
server_key_iv: "8a44588628402f6141c44bb23a70f0e2"
server_key_version: "encryptionkey@0"
I20260807 08:42:20.550886 30702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:20.565258 30702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:20.568397 30702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:20.568634 30702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:20.569443 30702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:20.569510 30702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:20.569548 30702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:20.569573 30702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 08:42:20 dist-test-slave-9zdj krb5kdc[30265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092140, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:20.579977 30702 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:43051
I20260807 08:42:20.580456 30702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:20.580559 30846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:43051 every 8 connection(s)
I20260807 08:42:20.583626 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.569999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:38439 (local address 127.28.108.190:34059)
0807 08:42:20.570079 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.570083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.570450 (+ 367us) server_negotiation.cc:407] Connection header received
0807 08:42:20.571457 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.571458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.571475 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.571554 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:20.572830 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.573370 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.575633 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.575868 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.575989 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.579356 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:20.579359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:20.579359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:20.579370 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:20.581471 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.582118 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.582120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.582120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.582219 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.582572 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.582574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.582575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.582734 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:20.582797 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.583444 (+ 647us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.583544 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 08:42:20.583896 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.569989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34059 (local address 127.28.108.131:38439)
0807 08:42:20.570222 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:20.570293 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:20.571175 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:20.571696 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:20.571703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:20.572254 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:20.572678 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.572689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.573514 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.573515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.575456 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.575462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.576024 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.576026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.576254 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.576982 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:20.576993 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:20.579193 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:20.581596 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.581603 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.581605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.581999 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.582323 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.582324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.582324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.582459 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.582908 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:20.582918 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:20.583265 (+ 347us) client_negotiation.cc:769] Sending connection context
0807 08:42:20.583380 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 08:42:20.583578 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":118,"threads_started":1}
I20260807 08:42:20.584622 30847 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34059
I20260807 08:42:20.584808 30847 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:20.585073 30847 heartbeater.cc:511] Master 127.28.108.190:34059 requested a full tablet report, sending...
I20260807 08:42:20.585891 30312 ts_manager.cc:194] Registered new tserver with Master: e1a47b3bef2545c380091109888f73ca (127.28.108.131:43051)
I20260807 08:42:20.586405 30312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:38439
I20260807 08:42:20.588754 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 30702
I20260807 08:42:20.589323 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:20.589563 29106 external_mini_cluster.cc:1428] Setting key 9dadba22fa8c46410ceb45a578744e0b
I20260807 08:42:20.590988 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:42:20.593981 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.586084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:50821 (local address 127.28.108.190:34059)
0807 08:42:20.586148 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.586151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.586383 (+ 232us) server_negotiation.cc:407] Connection header received
0807 08:42:20.586463 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.586464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.586476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.586527 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:20.587006 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.587507 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.588234 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.588465 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.588550 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.589936 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:20.589938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:20.589938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:20.589948 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:20.592201 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.592757 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.592759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.592760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.592847 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.593136 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.593137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.593137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.593365 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:20.593451 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.593699 (+ 248us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.593826 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260807 08:42:20.593984 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.586002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34059 (local address 127.28.108.131:50821)
0807 08:42:20.586217 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:20.586227 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:20.586405 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:20.586560 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:20.586561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:20.586830 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:20.586908 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.586910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.587609 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.587610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.588137 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.588139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.588569 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.588570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.588657 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.589172 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:20.589177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:20.589738 (+ 561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:20.592421 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.592422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.592423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.592648 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.592962 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:20.592963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:20.592964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:20.593048 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:20.593453 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:20.593454 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:20.593514 (+ 60us) client_negotiation.cc:769] Sending connection context
0807 08:42:20.593583 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 08:42:20.593816 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
Aug 07 08:42:20 dist-test-slave-9zdj krb5kdc[30265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092140, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 08:42:20 dist-test-slave-9zdj krb5kdc[30265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092140, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:20.624166 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.614563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36120 (local address 127.28.108.190:34059)
0807 08:42:20.614663 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.614665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.614700 (+ 35us) server_negotiation.cc:407] Connection header received
0807 08:42:20.614778 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.614779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.614790 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.614884 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:20.615632 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.616251 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.617170 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.617432 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.617519 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.620729 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:20.620731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:20.620731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:20.620740 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:20.622597 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.623107 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.623109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.623109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.623181 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:20.623517 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:20.623518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:20.623518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:20.623701 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:20.623803 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.623994 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.624084 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 08:42:20.626964 30312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:20.628830 30312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:20.637475 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.632319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:42223 (local address 127.0.0.1:48742)
0807 08:42:20.632685 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:20.632697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:20.632759 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:20.633224 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:20.633228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:20.633230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:20.633313 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.633315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.634523 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.634525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.635865 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.635869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.636984 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.636985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.637105 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.637122 (+ 17us) client_negotiation.cc:769] Sending connection context
0807 08:42:20.637243 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 08:42:20.637335 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":27,"thread_start_us":126,"threads_started":1}
I20260807 08:42:20.637485 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.632554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.28.108.130:42223)
0807 08:42:20.632954 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.632960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.632971 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:42:20.633012 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.633014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.633029 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.633118 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:20.633437 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.634421 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.635986 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.636872 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.636984 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.637051 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.637270 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.637322 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":85,"threads_started":1}
I20260807 08:42:20.637704 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.632578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.28.108.129:39009)
0807 08:42:20.632818 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.632825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.632837 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:42:20.632886 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.632889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.632909 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.633165 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:20.633445 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.634827 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.636145 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.636899 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.636981 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.637049 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.637478 (+ 429us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.637532 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":103,"threads_started":1}
I20260807 08:42:20.638173 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.632948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34186 (local address 127.28.108.131:43051)
0807 08:42:20.633239 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.633243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.633571 (+ 328us) server_negotiation.cc:407] Connection header received
0807 08:42:20.633652 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.633655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.633673 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.633766 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:20.633901 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.635166 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.636616 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.637445 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.637525 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.637583 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.637906 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.637955 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260807 08:42:20.637846 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.632222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:39009 (local address 127.0.0.1:33756)
0807 08:42:20.632516 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:20.632573 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:20.632655 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:20.633305 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:20.633306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:20.633308 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:20.633378 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.633380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.634942 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.634943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.636051 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.636053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.637089 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.637090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.637164 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.637169 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:20.637223 (+ 54us) client_negotiation.cc:240] Negotiation successful
0807 08:42:20.637326 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":11,"thread_start_us":105,"threads_started":1}
I20260807 08:42:20.637949 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.632496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:43051 (local address 127.0.0.1:34186)
0807 08:42:20.633456 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:20.633466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:20.633524 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:20.633773 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:20.633774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:20.633777 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:20.633831 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.633833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.635283 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.635284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.636492 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.636496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.637659 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.637661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.637741 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.637748 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 08:42:20.637808 (+ 60us) client_negotiation.cc:240] Negotiation successful
0807 08:42:20.637853 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":123,"threads_started":1}
I20260807 08:42:20.639632 30619 tablet_service.cc:1467] Processing CreateTablet for tablet bc77b36d0a4a42e2b9a5fa557a366211 (DEFAULT_TABLE table=test-table [id=c21ab1eda3704e0b89d1ffd2687b41df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:20.639706 30772 tablet_service.cc:1467] Processing CreateTablet for tablet bc77b36d0a4a42e2b9a5fa557a366211 (DEFAULT_TABLE table=test-table [id=c21ab1eda3704e0b89d1ffd2687b41df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:20.639658 30469 tablet_service.cc:1467] Processing CreateTablet for tablet bc77b36d0a4a42e2b9a5fa557a366211 (DEFAULT_TABLE table=test-table [id=c21ab1eda3704e0b89d1ffd2687b41df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:20.640106 30772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77b36d0a4a42e2b9a5fa557a366211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:20.640112 30469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77b36d0a4a42e2b9a5fa557a366211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:20.640110 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77b36d0a4a42e2b9a5fa557a366211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:20.643721 30870 tablet_bootstrap.cc:492] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: Bootstrap starting.
I20260807 08:42:20.644416 30872 tablet_bootstrap.cc:492] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b: Bootstrap starting.
I20260807 08:42:20.644704 30871 tablet_bootstrap.cc:492] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: Bootstrap starting.
I20260807 08:42:20.644986 30870 tablet_bootstrap.cc:654] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:20.645256 30872 tablet_bootstrap.cc:654] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:20.645406 30870 log.cc:824] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:20.645525 30871 tablet_bootstrap.cc:654] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:20.645702 30872 log.cc:824] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:20.646071 30871 log.cc:824] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:20.646397 30870 tablet_bootstrap.cc:492] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: No bootstrap required, opened a new log
I20260807 08:42:20.646489 30870 ts_tablet_manager.cc:1397] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260807 08:42:20.646685 30872 tablet_bootstrap.cc:492] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b: No bootstrap required, opened a new log
I20260807 08:42:20.646773 30872 ts_tablet_manager.cc:1397] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:42:20.646915 30871 tablet_bootstrap.cc:492] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: No bootstrap required, opened a new log
I20260807 08:42:20.646996 30871 ts_tablet_manager.cc:1397] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:42:20.648671 30872 raft_consensus.cc:348] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:20.648856 30872 raft_consensus.cc:374] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:20.648782 30870 raft_consensus.cc:348] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:20.648926 30872 raft_consensus.cc:729] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a5d10a37e543f89606dd49e74c3c4b, State: Initialized, Role: FOLLOWER
I20260807 08:42:20.649010 30870 raft_consensus.cc:374] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:20.649094 30870 raft_consensus.cc:729] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a47b3bef2545c380091109888f73ca, State: Initialized, Role: FOLLOWER
I20260807 08:42:20.649137 30872 consensus_queue.cc:260] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:20.649274 30870 consensus_queue.cc:260] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:20.649228 30871 raft_consensus.cc:348] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:20.649380 30871 raft_consensus.cc:374] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:20.649439 30871 raft_consensus.cc:729] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d51b176ade9445cb3cab8c7941aa2d4, State: Initialized, Role: FOLLOWER
I20260807 08:42:20.649479 30693 heartbeater.cc:503] Master 127.28.108.190:34059 was elected leader, sending a full tablet report...
I20260807 08:42:20.649567 30871 consensus_queue.cc:260] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:20.649796 30847 heartbeater.cc:503] Master 127.28.108.190:34059 was elected leader, sending a full tablet report...
I20260807 08:42:20.649885 30871 ts_tablet_manager.cc:1428] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:42:20.649971 30539 heartbeater.cc:503] Master 127.28.108.190:34059 was elected leader, sending a full tablet report...
I20260807 08:42:20.649727 30870 ts_tablet_manager.cc:1428] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 08:42:20.650274 30872 ts_tablet_manager.cc:1428] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260807 08:42:20.697465 30540 tablet.cc:2369] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:20.804117 30876 raft_consensus.cc:482] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:20.804365 30876 raft_consensus.cc:504] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:20.804689 30876 leader_election.cc:288] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d51b176ade9445cb3cab8c7941aa2d4 (127.28.108.129:39009), e1a47b3bef2545c380091109888f73ca (127.28.108.131:43051)
I20260807 08:42:20.809224 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.805012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:39009 (local address 127.28.108.130:43549)
0807 08:42:20.805112 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:20.805143 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:20.805214 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:20.805418 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:20.805419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:20.805421 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:20.805515 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.805517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.806636 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.806637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.807983 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.807986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.808978 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.808979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.809054 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.809057 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:20.809108 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 08:42:20.809137 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":59}
I20260807 08:42:20.809257 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.805550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:55195 (local address 127.28.108.131:43051)
0807 08:42:20.805610 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.805635 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.805720 (+ 85us) server_negotiation.cc:407] Connection header received
0807 08:42:20.805770 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.805771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.805781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.805846 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:20.805992 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.806796 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.808168 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.808800 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.808892 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.808903 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.809121 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.809175 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 08:42:20.809230 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.805397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:43051 (local address 127.28.108.130:55195)
0807 08:42:20.805659 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:20.805675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:20.805721 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:20.805863 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:20.805864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:20.805866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:20.805921 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.805922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.806884 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.806885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.808062 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:20.808064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.808904 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.808905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:20.809006 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.809010 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:20.809074 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 08:42:20.809122 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":156,"threads_started":1}
I20260807 08:42:20.809403 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.805100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:43549 (local address 127.28.108.129:39009)
0807 08:42:20.805194 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.805196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.805212 (+ 16us) server_negotiation.cc:407] Connection header received
0807 08:42:20.805275 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.805276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.805287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.805363 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:20.805662 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.806547 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.808148 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.808895 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.808967 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.808976 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.809138 (+ 162us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.809319 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 08:42:20.809989 30796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77b36d0a4a42e2b9a5fa557a366211" candidate_uuid: "28a5d10a37e543f89606dd49e74c3c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a47b3bef2545c380091109888f73ca" is_pre_election: true
I20260807 08:42:20.810005 30492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77b36d0a4a42e2b9a5fa557a366211" candidate_uuid: "28a5d10a37e543f89606dd49e74c3c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" is_pre_election: true
I20260807 08:42:20.810142 30796 raft_consensus.cc:2454] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a5d10a37e543f89606dd49e74c3c4b in term 0.
I20260807 08:42:20.810179 30492 raft_consensus.cc:2454] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a5d10a37e543f89606dd49e74c3c4b in term 0.
I20260807 08:42:20.810433 30568 leader_election.cc:302] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a5d10a37e543f89606dd49e74c3c4b, e1a47b3bef2545c380091109888f73ca; no voters:
I20260807 08:42:20.810757 30876 raft_consensus.cc:2781] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:20.810835 30876 raft_consensus.cc:482] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:20.810864 30876 raft_consensus.cc:3037] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:20.811679 30876 raft_consensus.cc:504] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:20.811851 30876 leader_election.cc:288] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [CANDIDATE]: Term 1 election: Requested vote from peers 0d51b176ade9445cb3cab8c7941aa2d4 (127.28.108.129:39009), e1a47b3bef2545c380091109888f73ca (127.28.108.131:43051)
I20260807 08:42:20.812039 30492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77b36d0a4a42e2b9a5fa557a366211" candidate_uuid: "28a5d10a37e543f89606dd49e74c3c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d51b176ade9445cb3cab8c7941aa2d4"
I20260807 08:42:20.812100 30796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77b36d0a4a42e2b9a5fa557a366211" candidate_uuid: "28a5d10a37e543f89606dd49e74c3c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a47b3bef2545c380091109888f73ca"
I20260807 08:42:20.812163 30492 raft_consensus.cc:3037] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:20.812211 30796 raft_consensus.cc:3037] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:20.813115 30492 raft_consensus.cc:2454] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a5d10a37e543f89606dd49e74c3c4b in term 1.
I20260807 08:42:20.813247 30796 raft_consensus.cc:2454] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a5d10a37e543f89606dd49e74c3c4b in term 1.
I20260807 08:42:20.813462 30568 leader_election.cc:302] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a5d10a37e543f89606dd49e74c3c4b, e1a47b3bef2545c380091109888f73ca; no voters:
I20260807 08:42:20.813591 30876 raft_consensus.cc:2781] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:20.813818 30876 raft_consensus.cc:686] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 1 LEADER]: Becoming Leader. State: Replica: 28a5d10a37e543f89606dd49e74c3c4b, State: Running, Role: LEADER
I20260807 08:42:20.813944 30876 consensus_queue.cc:237] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:20.814916 30311 catalog_manager.cc:5495] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a5d10a37e543f89606dd49e74c3c4b (127.28.108.130). New cstate: current_term: 1 leader_uuid: "28a5d10a37e543f89606dd49e74c3c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } health_report { overall_health: HEALTHY } } }
I20260807 08:42:20.830616 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:20.826912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.28.108.130:42223)
0807 08:42:20.826979 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:20.826981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:20.827005 (+ 24us) server_negotiation.cc:407] Connection header received
0807 08:42:20.827140 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:20.827141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:20.827160 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:20.827229 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:20.827524 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.828064 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.829153 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:20.829358 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:20.829422 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:20.829837 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:20.829968 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:20.830037 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:20.830143 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 08:42:20.830167 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260807 08:42:20.832058 30848 tablet.cc:2369] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:20.833065 30796 raft_consensus.cc:1261] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca [term 1 FOLLOWER]: Refusing update from remote peer 28a5d10a37e543f89606dd49e74c3c4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:20.833096 30492 raft_consensus.cc:1261] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 1 FOLLOWER]: Refusing update from remote peer 28a5d10a37e543f89606dd49e74c3c4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:20.833491 30876 consensus_queue.cc:1035] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:20.833765 30880 consensus_queue.cc:1035] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:20.836196 30883 mvcc.cc:205] Tried to move back new op lower bound from 7315833408848281600 to 7315833408774717440. Current Snapshot: MvccSnapshot[applied={T|T < 7315833408848281600}]
I20260807 08:42:20.836921 30884 mvcc.cc:205] Tried to move back new op lower bound from 7315833408848281600 to 7315833408774717440. Current Snapshot: MvccSnapshot[applied={T|T < 7315833408848281600}]
I20260807 08:42:20.838007 30886 mvcc.cc:205] Tried to move back new op lower bound from 7315833408848281600 to 7315833408774717440. Current Snapshot: MvccSnapshot[applied={T|T < 7315833408848281600}]
I20260807 08:42:20.841145 30311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:20.841259 30311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:20.842450 30311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e: Sending DeleteTablet for 3 replicas of tablet bc77b36d0a4a42e2b9a5fa557a366211
I20260807 08:42:20.843019 30469 tablet_service.cc:1514] Processing DeleteTablet for tablet bc77b36d0a4a42e2b9a5fa557a366211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:20 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:33756
I20260807 08:42:20.843061 30772 tablet_service.cc:1514] Processing DeleteTablet for tablet bc77b36d0a4a42e2b9a5fa557a366211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:20 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:34186
I20260807 08:42:20.843158 30619 tablet_service.cc:1514] Processing DeleteTablet for tablet bc77b36d0a4a42e2b9a5fa557a366211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:20 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:48742
I20260807 08:42:20.843420 30894 tablet_replica.cc:330] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: stopping tablet replica
I20260807 08:42:20.843365 30311 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:20.843568 30311 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:20.843601 30894 raft_consensus.cc:2229] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:20.843751 30894 raft_consensus.cc:2258] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:20.844414 30894 ts_tablet_manager.cc:1905] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:20.845003 30469 tablet_service.cc:1467] Processing CreateTablet for tablet f5e14263aec743be9427f74ff5130a93 (DEFAULT_TABLE table=test-table [id=4f8045001f9740338636a371f9f0685d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:20.845032 30619 tablet_service.cc:1467] Processing CreateTablet for tablet f5e14263aec743be9427f74ff5130a93 (DEFAULT_TABLE table=test-table [id=4f8045001f9740338636a371f9f0685d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:20.845158 30469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e14263aec743be9427f74ff5130a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:20.845175 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e14263aec743be9427f74ff5130a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:20.846391 30772 tablet_service.cc:1467] Processing CreateTablet for tablet f5e14263aec743be9427f74ff5130a93 (DEFAULT_TABLE table=test-table [id=4f8045001f9740338636a371f9f0685d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:20.846537 30772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e14263aec743be9427f74ff5130a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:20.846817 30872 tablet_bootstrap.cc:492] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b: Bootstrap starting.
I20260807 08:42:20.847311 30872 tablet_bootstrap.cc:654] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:20.847728 30870 tablet_bootstrap.cc:492] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca: Bootstrap starting.
I20260807 08:42:20.847841 30894 ts_tablet_manager.cc:1918] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:20.847988 30894 log.cc:1196] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/wal/wals/bc77b36d0a4a42e2b9a5fa557a366211
I20260807 08:42:20.848181 30870 tablet_bootstrap.cc:654] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:20.848353 30894 ts_tablet_manager.cc:1939] T bc77b36d0a4a42e2b9a5fa557a366211 P 0d51b176ade9445cb3cab8c7941aa2d4: Deleting consensus metadata
I20260807 08:42:20.849241 30871 tablet_bootstrap.cc:492] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4: Bootstrap starting.
I20260807 08:42:20.849440 30872 tablet_bootstrap.cc:492] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b: No bootstrap required, opened a new log
I20260807 08:42:20.849509 30872 ts_tablet_manager.cc:1397] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 08:42:20.849735 30871 tablet_bootstrap.cc:654] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:20.849731 30895 tablet_replica.cc:330] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: stopping tablet replica
I20260807 08:42:20.849759 30872 raft_consensus.cc:348] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } }
I20260807 08:42:20.849854 30872 raft_consensus.cc:374] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:20.849876 30872 raft_consensus.cc:729] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a5d10a37e543f89606dd49e74c3c4b, State: Initialized, Role: FOLLOWER
I20260807 08:42:20.849876 30895 raft_consensus.cc:2229] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:20.849938 30872 consensus_queue.cc:260] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } }
I20260807 08:42:20.850039 30872 ts_tablet_manager.cc:1428] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:20.850152 30297 catalog_manager.cc:4841] TS 0d51b176ade9445cb3cab8c7941aa2d4 (127.28.108.129:39009): tablet bc77b36d0a4a42e2b9a5fa557a366211 (table test-table [id=c21ab1eda3704e0b89d1ffd2687b41df]) successfully deleted
I20260807 08:42:20.850920 30871 tablet_bootstrap.cc:492] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4: No bootstrap required, opened a new log
I20260807 08:42:20.850030 30895 raft_consensus.cc:2258] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:20.850559 30870 tablet_bootstrap.cc:492] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca: No bootstrap required, opened a new log
I20260807 08:42:20.850979 30871 ts_tablet_manager.cc:1397] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 08:42:20.850992 30870 ts_tablet_manager.cc:1397] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 08:42:20.851181 30870 raft_consensus.cc:348] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } }
I20260807 08:42:20.851270 30870 raft_consensus.cc:374] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:20.851293 30870 raft_consensus.cc:729] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a47b3bef2545c380091109888f73ca, State: Initialized, Role: FOLLOWER
I20260807 08:42:20.851342 30870 consensus_queue.cc:260] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } }
I20260807 08:42:20.851375 30871 raft_consensus.cc:348] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } }
I20260807 08:42:20.851430 30870 ts_tablet_manager.cc:1428] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:20.851450 30871 raft_consensus.cc:374] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:20.851475 30871 raft_consensus.cc:729] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d51b176ade9445cb3cab8c7941aa2d4, State: Initialized, Role: FOLLOWER
I20260807 08:42:20.851527 30871 consensus_queue.cc:260] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } }
I20260807 08:42:20.851642 30871 ts_tablet_manager.cc:1428] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:20.854045 30895 ts_tablet_manager.cc:1905] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:20.856053 30896 tablet_replica.cc:330] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b: stopping tablet replica
I20260807 08:42:20.856213 30896 raft_consensus.cc:2229] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:20.856343 30896 raft_consensus.cc:2258] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:20.856549 30895 ts_tablet_manager.cc:1918] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:20.856649 30895 log.cc:1196] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/bc77b36d0a4a42e2b9a5fa557a366211
I20260807 08:42:20.856985 30896 ts_tablet_manager.cc:1905] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:20.857072 30895 ts_tablet_manager.cc:1939] T bc77b36d0a4a42e2b9a5fa557a366211 P e1a47b3bef2545c380091109888f73ca: Deleting consensus metadata
I20260807 08:42:20.857671 30297 catalog_manager.cc:4841] TS e1a47b3bef2545c380091109888f73ca (127.28.108.131:43051): tablet bc77b36d0a4a42e2b9a5fa557a366211 (table test-table [id=c21ab1eda3704e0b89d1ffd2687b41df]) successfully deleted
I20260807 08:42:20.858892 30896 ts_tablet_manager.cc:1918] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:20.858987 30896 log.cc:1196] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/bc77b36d0a4a42e2b9a5fa557a366211
I20260807 08:42:20.859273 30896 ts_tablet_manager.cc:1939] T bc77b36d0a4a42e2b9a5fa557a366211 P 28a5d10a37e543f89606dd49e74c3c4b: Deleting consensus metadata
I20260807 08:42:20.859874 30296 catalog_manager.cc:4841] TS 28a5d10a37e543f89606dd49e74c3c4b (127.28.108.130:42223): tablet bc77b36d0a4a42e2b9a5fa557a366211 (table test-table [id=c21ab1eda3704e0b89d1ffd2687b41df]) successfully deleted
W20260807 08:42:20.885442 30694 tablet.cc:2369] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:21.255134 30877 raft_consensus.cc:482] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:21.255275 30877 raft_consensus.cc:504] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } }
I20260807 08:42:21.255779 30877 leader_election.cc:288] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d51b176ade9445cb3cab8c7941aa2d4 (127.28.108.129:39009), 28a5d10a37e543f89606dd49e74c3c4b (127.28.108.130:42223)
I20260807 08:42:21.259786 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.256041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:42223 (local address 127.28.108.131:54899)
0807 08:42:21.256332 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:21.256342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:21.256471 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:21.256667 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:21.256668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:21.256670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:21.257042 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:21.257045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.258014 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.258015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:21.258827 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:21.258829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.259477 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.259479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:21.259555 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.259560 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:21.259612 (+ 52us) client_negotiation.cc:240] Negotiation successful
0807 08:42:21.259651 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":23,"thread_start_us":63,"threads_started":1}
I20260807 08:42:21.259784 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.256129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:54899 (local address 127.28.108.130:42223)
0807 08:42:21.256236 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:21.256238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:21.256485 (+ 247us) server_negotiation.cc:407] Connection header received
0807 08:42:21.256535 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:21.256536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:21.256572 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:21.256641 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:21.257187 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.257903 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.258991 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.259367 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.259434 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.259444 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:21.259648 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 08:42:21.259711 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 08:42:21.259887 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.256131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:35435 (local address 127.28.108.129:39009)
0807 08:42:21.256178 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:21.256181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:21.256439 (+ 258us) server_negotiation.cc:407] Connection header received
0807 08:42:21.256503 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:21.256504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:21.256515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:21.256610 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:21.257022 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.257690 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.258948 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.259434 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.259518 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.259528 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:21.259750 (+ 222us) server_negotiation.cc:299] Negotiation successful
0807 08:42:21.259807 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 08:42:21.259935 30901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.256039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:39009 (local address 127.28.108.131:35435)
0807 08:42:21.256312 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:21.256325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:21.256414 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:21.256602 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:21.256603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:21.256606 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:21.256745 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:21.256747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.257783 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.257785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:21.258815 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:21.258819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.259553 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.259555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:21.259648 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.259654 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:42:21.259715 (+ 61us) client_negotiation.cc:240] Negotiation successful
0807 08:42:21.259762 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":3,"thread_start_us":222,"threads_started":1}
I20260807 08:42:21.260401 30643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e14263aec743be9427f74ff5130a93" candidate_uuid: "e1a47b3bef2545c380091109888f73ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a5d10a37e543f89606dd49e74c3c4b" is_pre_election: true
I20260807 08:42:21.260529 30492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e14263aec743be9427f74ff5130a93" candidate_uuid: "e1a47b3bef2545c380091109888f73ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" is_pre_election: true
I20260807 08:42:21.260627 30643 raft_consensus.cc:2454] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a47b3bef2545c380091109888f73ca in term 0.
I20260807 08:42:21.260655 30492 raft_consensus.cc:2454] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a47b3bef2545c380091109888f73ca in term 0.
I20260807 08:42:21.260859 30721 leader_election.cc:302] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d51b176ade9445cb3cab8c7941aa2d4, e1a47b3bef2545c380091109888f73ca; no voters:
I20260807 08:42:21.261034 30877 raft_consensus.cc:2781] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:21.261085 30877 raft_consensus.cc:482] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:21.261101 30877 raft_consensus.cc:3037] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:21.261950 30877 raft_consensus.cc:504] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } }
I20260807 08:42:21.262147 30877 leader_election.cc:288] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [CANDIDATE]: Term 1 election: Requested vote from peers 0d51b176ade9445cb3cab8c7941aa2d4 (127.28.108.129:39009), 28a5d10a37e543f89606dd49e74c3c4b (127.28.108.130:42223)
I20260807 08:42:21.262337 30492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e14263aec743be9427f74ff5130a93" candidate_uuid: "e1a47b3bef2545c380091109888f73ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d51b176ade9445cb3cab8c7941aa2d4"
I20260807 08:42:21.262344 30643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e14263aec743be9427f74ff5130a93" candidate_uuid: "e1a47b3bef2545c380091109888f73ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a5d10a37e543f89606dd49e74c3c4b"
I20260807 08:42:21.262426 30492 raft_consensus.cc:3037] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:21.262428 30643 raft_consensus.cc:3037] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:21.263485 30492 raft_consensus.cc:2454] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a47b3bef2545c380091109888f73ca in term 1.
I20260807 08:42:21.263509 30643 raft_consensus.cc:2454] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a47b3bef2545c380091109888f73ca in term 1.
I20260807 08:42:21.263747 30721 leader_election.cc:302] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d51b176ade9445cb3cab8c7941aa2d4, e1a47b3bef2545c380091109888f73ca; no voters:
I20260807 08:42:21.263873 30877 raft_consensus.cc:2781] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:21.264002 30877 raft_consensus.cc:686] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 1 LEADER]: Becoming Leader. State: Replica: e1a47b3bef2545c380091109888f73ca, State: Running, Role: LEADER
I20260807 08:42:21.264093 30877 consensus_queue.cc:237] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } }
I20260807 08:42:21.265004 30311 catalog_manager.cc:5495] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca reported cstate change: term changed from 0 to 1, leader changed from <none> to e1a47b3bef2545c380091109888f73ca (127.28.108.131). New cstate: current_term: 1 leader_uuid: "e1a47b3bef2545c380091109888f73ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } health_report { overall_health: HEALTHY } } }
Aug 07 08:42:21 dist-test-slave-9zdj krb5kdc[30265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:21.323443 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.314449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34059 (local address 127.0.0.1:36136)
0807 08:42:21.314646 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:21.314660 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:21.314753 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:21.314962 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:21.314963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:21.315317 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:21.315442 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:21.315446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.316436 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.316436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:21.317081 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:21.317083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.317582 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.317583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:21.317753 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.318410 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:21.318415 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:21.320397 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:21.322266 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:21.322275 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:21.322278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:21.322554 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:21.322807 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:21.322808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:21.322809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:21.322885 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:21.323135 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:21.323143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:21.323194 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 08:42:21.323255 (+ 61us) client_negotiation.cc:240] Negotiation successful
0807 08:42:21.323360 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260807 08:42:21.323722 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.314557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36136 (local address 127.28.108.190:34059)
0807 08:42:21.314810 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:21.314812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:21.314822 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:21.314855 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:21.314856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:21.314867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:21.314940 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:21.315553 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.316288 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.317190 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.317474 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.317544 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.320515 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:21.320516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:21.320517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:21.320524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:21.322165 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:21.322672 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:21.322673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:21.322673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:21.322737 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:21.322948 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:21.322949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:21.322949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:21.323071 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:21.323127 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:21.323468 (+ 341us) server_negotiation.cc:299] Negotiation successful
0807 08:42:21.323617 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":119,"threads_started":1}
I20260807 08:42:21.326488 30311 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:36136:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 08:42:21.326861 30311 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:21.328856 30469 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbd242906ac4a0b9d052c054391487a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6092b1e719e48608539fd4a5693224d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 08:42:21.329129 30469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbd242906ac4a0b9d052c054391487a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:21.328867 30619 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbd242906ac4a0b9d052c054391487a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6092b1e719e48608539fd4a5693224d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 08:42:21.330799 30871 tablet_bootstrap.cc:492] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4: Bootstrap starting.
I20260807 08:42:21.330941 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbd242906ac4a0b9d052c054391487a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:21.331326 30871 tablet_bootstrap.cc:654] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:21.328857 30772 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbd242906ac4a0b9d052c054391487a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6092b1e719e48608539fd4a5693224d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 08:42:21.331555 30772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbd242906ac4a0b9d052c054391487a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:21.332120 30871 tablet_bootstrap.cc:492] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4: No bootstrap required, opened a new log
I20260807 08:42:21.332204 30871 ts_tablet_manager.cc:1397] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:21.332378 30871 raft_consensus.cc:348] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:21.332479 30871 raft_consensus.cc:374] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:21.332507 30871 raft_consensus.cc:729] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d51b176ade9445cb3cab8c7941aa2d4, State: Initialized, Role: FOLLOWER
I20260807 08:42:21.332556 30871 consensus_queue.cc:260] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:21.332672 30871 ts_tablet_manager.cc:1428] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:21.332767 30878 tablet_replica.cc:439] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } }
I20260807 08:42:21.332898 30878 tablet_replica.cc:442] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:21.333390 30870 tablet_bootstrap.cc:492] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: Bootstrap starting.
I20260807 08:42:21.333943 30870 tablet_bootstrap.cc:654] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:21.334569 30872 tablet_bootstrap.cc:492] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b: Bootstrap starting.
I20260807 08:42:21.334774 30870 tablet_bootstrap.cc:492] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: No bootstrap required, opened a new log
I20260807 08:42:21.334836 30870 ts_tablet_manager.cc:1397] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:21.334969 30872 tablet_bootstrap.cc:654] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:21.335023 30870 raft_consensus.cc:348] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:21.335093 30870 raft_consensus.cc:374] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:21.335108 30870 raft_consensus.cc:729] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a47b3bef2545c380091109888f73ca, State: Initialized, Role: FOLLOWER
I20260807 08:42:21.335170 30870 consensus_queue.cc:260] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:21.335265 30870 ts_tablet_manager.cc:1428] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:21.335419 30877 tablet_replica.cc:439] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } }
I20260807 08:42:21.335543 30877 tablet_replica.cc:442] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:21.335577 30872 tablet_bootstrap.cc:492] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b: No bootstrap required, opened a new log
I20260807 08:42:21.335635 30872 ts_tablet_manager.cc:1397] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:21.335768 30872 raft_consensus.cc:348] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:21.335850 30872 raft_consensus.cc:374] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:21.335894 30872 raft_consensus.cc:729] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a5d10a37e543f89606dd49e74c3c4b, State: Initialized, Role: FOLLOWER
I20260807 08:42:21.335951 30872 consensus_queue.cc:260] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:21.336091 30872 ts_tablet_manager.cc:1428] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 08:42:21.336128 30880 tablet_replica.cc:439] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } }
I20260807 08:42:21.336236 30880 tablet_replica.cc:442] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:21.345527 30877 raft_consensus.cc:482] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:21.345713 30877 raft_consensus.cc:504] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:21.345904 30877 leader_election.cc:288] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d51b176ade9445cb3cab8c7941aa2d4 (127.28.108.129:39009), 28a5d10a37e543f89606dd49e74c3c4b (127.28.108.130:42223)
I20260807 08:42:21.346139 30643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbd242906ac4a0b9d052c054391487a" candidate_uuid: "e1a47b3bef2545c380091109888f73ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a5d10a37e543f89606dd49e74c3c4b" is_pre_election: true
I20260807 08:42:21.346166 30492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbd242906ac4a0b9d052c054391487a" candidate_uuid: "e1a47b3bef2545c380091109888f73ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" is_pre_election: true
I20260807 08:42:21.346263 30643 raft_consensus.cc:2454] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a47b3bef2545c380091109888f73ca in term 0.
I20260807 08:42:21.346304 30492 raft_consensus.cc:2454] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a47b3bef2545c380091109888f73ca in term 0.
I20260807 08:42:21.346457 30721 leader_election.cc:302] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d51b176ade9445cb3cab8c7941aa2d4, e1a47b3bef2545c380091109888f73ca; no voters:
I20260807 08:42:21.346571 30877 raft_consensus.cc:2781] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:21.346617 30877 raft_consensus.cc:482] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:21.346643 30877 raft_consensus.cc:3037] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:21.347453 30877 raft_consensus.cc:504] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:21.347621 30877 leader_election.cc:288] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [CANDIDATE]: Term 1 election: Requested vote from peers 0d51b176ade9445cb3cab8c7941aa2d4 (127.28.108.129:39009), 28a5d10a37e543f89606dd49e74c3c4b (127.28.108.130:42223)
I20260807 08:42:21.347855 30643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbd242906ac4a0b9d052c054391487a" candidate_uuid: "e1a47b3bef2545c380091109888f73ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a5d10a37e543f89606dd49e74c3c4b"
I20260807 08:42:21.347828 30492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbd242906ac4a0b9d052c054391487a" candidate_uuid: "e1a47b3bef2545c380091109888f73ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d51b176ade9445cb3cab8c7941aa2d4"
I20260807 08:42:21.347952 30643 raft_consensus.cc:3037] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:21.347952 30492 raft_consensus.cc:3037] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:21.348960 30492 raft_consensus.cc:2454] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a47b3bef2545c380091109888f73ca in term 1.
I20260807 08:42:21.349052 30643 raft_consensus.cc:2454] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a47b3bef2545c380091109888f73ca in term 1.
I20260807 08:42:21.349318 30721 leader_election.cc:302] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d51b176ade9445cb3cab8c7941aa2d4, e1a47b3bef2545c380091109888f73ca; no voters:
I20260807 08:42:21.349452 30877 raft_consensus.cc:2781] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:21.349565 30877 raft_consensus.cc:686] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [term 1 LEADER]: Becoming Leader. State: Replica: e1a47b3bef2545c380091109888f73ca, State: Running, Role: LEADER
I20260807 08:42:21.349660 30877 consensus_queue.cc:237] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } }
I20260807 08:42:21.350118 30902 tablet_replica.cc:439] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: TxnStatusTablet state changed. Reason: New leader e1a47b3bef2545c380091109888f73ca. Latest consensus state: current_term: 1 leader_uuid: "e1a47b3bef2545c380091109888f73ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } }
I20260807 08:42:21.350238 30902 tablet_replica.cc:442] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: This TxnStatusTablet replica's current role is: LEADER
I20260807 08:42:21.350598 30313 catalog_manager.cc:5495] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca reported cstate change: term changed from 0 to 1, leader changed from <none> to e1a47b3bef2545c380091109888f73ca (127.28.108.131). New cstate: current_term: 1 leader_uuid: "e1a47b3bef2545c380091109888f73ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } health_report { overall_health: UNKNOWN } } }
I20260807 08:42:21.351881 30915 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 08:42:21.352087 30915 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 08:42:21.361598 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.357888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:43051 (local address 127.0.0.1:34198)
0807 08:42:21.358002 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:21.358035 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:21.358097 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:21.358464 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:21.358466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:21.358468 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:21.358532 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:21.358535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.359669 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.359671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:21.360360 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:21.360362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.360878 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.360879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:21.360970 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.361001 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:21.361359 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:21.361374 (+ 15us) client_negotiation.cc:769] Sending connection context
0807 08:42:21.361438 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 08:42:21.361518 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 08:42:21.361595 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.357953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.28.108.131:43051)
0807 08:42:21.358087 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:21.358089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:21.358100 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:42:21.358314 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:21.358315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:21.358339 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:21.358438 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:21.358631 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.359550 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.360468 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.360765 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.360864 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.361130 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:21.361274 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:21.361378 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:21.361445 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 08:42:21.361474 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260807 08:42:21.363698 30643 raft_consensus.cc:1261] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b [term 1 FOLLOWER]: Refusing update from remote peer e1a47b3bef2545c380091109888f73ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:21.363675 30492 raft_consensus.cc:1261] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4 [term 1 FOLLOWER]: Refusing update from remote peer e1a47b3bef2545c380091109888f73ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:21.364039 30877 consensus_queue.cc:1035] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:21.364171 30902 consensus_queue.cc:1035] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:21.364745 30878 tablet_replica.cc:439] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4: TxnStatusTablet state changed. Reason: New leader e1a47b3bef2545c380091109888f73ca. Latest consensus state: current_term: 1 leader_uuid: "e1a47b3bef2545c380091109888f73ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } }
I20260807 08:42:21.364857 30878 tablet_replica.cc:442] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:21.365072 30880 tablet_replica.cc:439] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b: TxnStatusTablet state changed. Reason: New leader e1a47b3bef2545c380091109888f73ca. Latest consensus state: current_term: 1 leader_uuid: "e1a47b3bef2545c380091109888f73ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } }
I20260807 08:42:21.365226 30880 tablet_replica.cc:442] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:21.365988 30877 tablet_replica.cc:439] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1a47b3bef2545c380091109888f73ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } }
I20260807 08:42:21.366097 30877 tablet_replica.cc:442] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: This TxnStatusTablet replica's current role is: LEADER
I20260807 08:42:21.366634 30880 tablet_replica.cc:439] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1a47b3bef2545c380091109888f73ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } }
I20260807 08:42:21.366730 30880 tablet_replica.cc:442] T 7cbd242906ac4a0b9d052c054391487a P 28a5d10a37e543f89606dd49e74c3c4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:21.367501 30902 tablet_replica.cc:439] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1a47b3bef2545c380091109888f73ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } }
I20260807 08:42:21.367601 30902 tablet_replica.cc:442] T 7cbd242906ac4a0b9d052c054391487a P e1a47b3bef2545c380091109888f73ca: This TxnStatusTablet replica's current role is: LEADER
I20260807 08:42:21.368018 30878 tablet_replica.cc:439] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1a47b3bef2545c380091109888f73ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 } } peers { permanent_uuid: "e1a47b3bef2545c380091109888f73ca" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43051 } } peers { permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 } } }
I20260807 08:42:21.368127 30878 tablet_replica.cc:442] T 7cbd242906ac4a0b9d052c054391487a P 0d51b176ade9445cb3cab8c7941aa2d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:21.372939 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.369190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.28.108.131:43051)
0807 08:42:21.369454 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:21.369456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:21.369467 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:42:21.369493 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:21.369494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:21.369513 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:21.369569 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:21.369928 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.370496 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.371902 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.372148 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.372376 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.372592 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:21.372631 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:21.372701 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:21.372788 (+ 87us) server_negotiation.cc:299] Negotiation successful
0807 08:42:21.372818 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260807 08:42:21.380390 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.376505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:48149 (local address 127.28.108.131:43051)
0807 08:42:21.376556 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:21.376559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:21.376972 (+ 413us) server_negotiation.cc:407] Connection header received
0807 08:42:21.377049 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:21.377050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:21.377060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:21.377132 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:21.377320 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.378137 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.379385 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.379724 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.379795 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.380015 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:21.380052 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:21.380111 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:21.380234 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 08:42:21.380261 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:42:21.380401 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.376496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:43051 (local address 127.28.108.131:48149)
0807 08:42:21.376658 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:21.376670 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:21.376978 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:21.377134 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:21.377135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:21.377136 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:21.377248 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:21.377250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.378233 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.378234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:21.379296 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:21.379300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.379798 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.379798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:21.379907 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.379927 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:21.380153 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:21.380156 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:21.380221 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 08:42:21.380285 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260807 08:42:21.383293 30492 raft_consensus.cc:1261] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 1 FOLLOWER]: Refusing update from remote peer e1a47b3bef2545c380091109888f73ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:21.383288 30643 raft_consensus.cc:1261] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b [term 1 FOLLOWER]: Refusing update from remote peer e1a47b3bef2545c380091109888f73ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:21.383553 30902 consensus_queue.cc:1035] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a5d10a37e543f89606dd49e74c3c4b" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:21.383718 30877 consensus_queue.cc:1035] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d51b176ade9445cb3cab8c7941aa2d4" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:21.402877 30313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:21.403021 30313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:21.403939 30313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2230b2d94d842608bd4fe624b1dc24e: Sending DeleteTablet for 3 replicas of tablet f5e14263aec743be9427f74ff5130a93
I20260807 08:42:21.404264 30619 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e14263aec743be9427f74ff5130a93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:21 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:48742
I20260807 08:42:21.404335 30771 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e14263aec743be9427f74ff5130a93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:21 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:34186
I20260807 08:42:21.404418 30469 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e14263aec743be9427f74ff5130a93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:21 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:33756
I20260807 08:42:21.404672 30939 tablet_replica.cc:330] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca: stopping tablet replica
I20260807 08:42:21.404742 30939 raft_consensus.cc:2229] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:21.404852 30939 raft_consensus.cc:2258] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:21.405164 30939 ts_tablet_manager.cc:1905] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:21.405937 30940 tablet_replica.cc:330] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4: stopping tablet replica
I20260807 08:42:21.406046 30940 raft_consensus.cc:2229] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:21.406101 30940 raft_consensus.cc:2258] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:21.406183 30938 tablet_replica.cc:330] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b: stopping tablet replica
I20260807 08:42:21.406293 30938 raft_consensus.cc:2229] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:21.406352 30938 raft_consensus.cc:2258] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:21.406702 30938 ts_tablet_manager.cc:1905] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:21.407361 30939 ts_tablet_manager.cc:1918] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:21.407429 30939 log.cc:1196] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/f5e14263aec743be9427f74ff5130a93
I20260807 08:42:21.407716 30939 ts_tablet_manager.cc:1939] T f5e14263aec743be9427f74ff5130a93 P e1a47b3bef2545c380091109888f73ca: Deleting consensus metadata
I20260807 08:42:21.408162 30297 catalog_manager.cc:4841] TS e1a47b3bef2545c380091109888f73ca (127.28.108.131:43051): tablet f5e14263aec743be9427f74ff5130a93 (table test-table [id=4f8045001f9740338636a371f9f0685d]) successfully deleted
I20260807 08:42:21.408478 30938 ts_tablet_manager.cc:1918] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:21.408565 30938 log.cc:1196] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/f5e14263aec743be9427f74ff5130a93
I20260807 08:42:21.408814 30938 ts_tablet_manager.cc:1939] T f5e14263aec743be9427f74ff5130a93 P 28a5d10a37e543f89606dd49e74c3c4b: Deleting consensus metadata
I20260807 08:42:21.409134 30940 ts_tablet_manager.cc:1905] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:21.409689 30296 catalog_manager.cc:4841] TS 28a5d10a37e543f89606dd49e74c3c4b (127.28.108.130:42223): tablet f5e14263aec743be9427f74ff5130a93 (table test-table [id=4f8045001f9740338636a371f9f0685d]) successfully deleted
I20260807 08:42:21.411104 30940 ts_tablet_manager.cc:1918] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:21.411178 30940 log.cc:1196] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786092133523677-29106-0/minicluster-data/ts-0/wal/wals/f5e14263aec743be9427f74ff5130a93
I20260807 08:42:21.411429 30940 ts_tablet_manager.cc:1939] T f5e14263aec743be9427f74ff5130a93 P 0d51b176ade9445cb3cab8c7941aa2d4: Deleting consensus metadata
I20260807 08:42:21.411864 30297 catalog_manager.cc:4841] TS 0d51b176ade9445cb3cab8c7941aa2d4 (127.28.108.129:39009): tablet f5e14263aec743be9427f74ff5130a93 (table test-table [id=4f8045001f9740338636a371f9f0685d]) successfully deleted
Aug 07 08:42:21 dist-test-slave-9zdj krb5kdc[30265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092140, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
I20260807 08:42:21.422345 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.405740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.28.108.129:39009)
0807 08:42:21.405801 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:21.405804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:21.411144 (+ 5340us) server_negotiation.cc:407] Connection header received
0807 08:42:21.411266 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:21.411268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:21.411285 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:21.411326 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:21.412250 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.413062 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.414203 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.414516 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.414601 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.418297 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:21.418316 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:21.418317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:21.418327 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:21.420628 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:21.421165 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:21.421175 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:21.421176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:21.421282 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:21.421595 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:21.421596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:21.421597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:21.421863 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:21.422024 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:21.422124 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 08:42:21.422227 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260807 08:42:21.422925 30533 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33772
I20260807 08:42:21.431890 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:21.424705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.28.108.190:34059)
0807 08:42:21.424831 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:21.424834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:21.424864 (+ 30us) server_negotiation.cc:407] Connection header received
0807 08:42:21.424949 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:21.424950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:21.424969 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:21.425030 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:21.425774 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.426298 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.427029 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:21.427248 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:21.427328 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:21.428704 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:21.428706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:21.428706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:21.428715 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:21.430402 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:21.430814 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:21.430816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:21.430816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:21.430893 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:21.431122 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:21.431123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:21.431124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:21.431269 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:21.431472 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:21.431643 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 08:42:21.431751 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260807 08:42:21.432381 30313 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36146
I20260807 08:42:21.433241 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 30396
I20260807 08:42:21.445894 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 30549
I20260807 08:42:21.458853 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 30702
I20260807 08:42:21.467185 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 30278
2026-08-07T08:42:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2979 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:21 dist-test-slave-9zdj krb5kdc[30948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:21 dist-test-slave-9zdj krb5kdc[30948](info): set up 2 sockets
Aug 07 08:42:21 dist-test-slave-9zdj krb5kdc[30948](info): commencing operation
krb5kdc: starting...
W20260807 08:42:22.616037 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:22 dist-test-slave-9zdj krb5kdc[30948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:22.770087 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:36229
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:36229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:22.869776 30964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:22.870113 30964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:22.870163 30964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:22.871701 30964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:22.871811 30964 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 08:42:22.871842 30964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:22.871868 30964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:22.871901 30964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:22.871932 30964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:22.874006 30964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:36229
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:22.874437 30964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:22.874756 30964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:22.880882 30970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:22.880854 30973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:22.880869 30969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:22.881132 30964 server_base.cc:1034] running on GCE node
I20260807 08:42:22.881449 30964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:22.881861 30964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:22.883046 30964 hybrid_clock.cc:648] HybridClock initialized: now 1786092142883026 us; error 34 us; skew 500 ppm
Aug 07 08:42:22 dist-test-slave-9zdj krb5kdc[30948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:22.886262 30964 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:22.886855 30964 webserver.cc:460] Webserver started at http://127.28.108.190:35401/ using document root <none> and password file <none>
I20260807 08:42:22.887125 30964 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:22.887207 30964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:22.887369 30964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:22.888634 30964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "5d7d7f7d04af4c81854181e0863f4fdb"
format_stamp: "Formatted at 2026-08-07 08:42:22 on dist-test-slave-9zdj"
server_key: "6699d1379d97ad593524bae2b8cdd08c"
server_key_iv: "2da1f13d37e95c7804c6f7079236e485"
server_key_version: "encryptionkey@0"
I20260807 08:42:22.889036 30964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "5d7d7f7d04af4c81854181e0863f4fdb"
format_stamp: "Formatted at 2026-08-07 08:42:22 on dist-test-slave-9zdj"
server_key: "6699d1379d97ad593524bae2b8cdd08c"
server_key_iv: "2da1f13d37e95c7804c6f7079236e485"
server_key_version: "encryptionkey@0"
I20260807 08:42:22.890650 30964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 08:42:22.891713 30980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:22.892112 30964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:22.892388 30964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "5d7d7f7d04af4c81854181e0863f4fdb"
format_stamp: "Formatted at 2026-08-07 08:42:22 on dist-test-slave-9zdj"
server_key: "6699d1379d97ad593524bae2b8cdd08c"
server_key_iv: "2da1f13d37e95c7804c6f7079236e485"
server_key_version: "encryptionkey@0"
I20260807 08:42:22.892488 30964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:22.919713 30964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:22.922520 30964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:22.922709 30964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:22.929219 30964 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:36229
I20260807 08:42:22.929323 31047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:36229 every 8 connection(s)
I20260807 08:42:22.929766 30964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:22.930534 31049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:22.933140 31049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb: Bootstrap starting.
I20260807 08:42:22.933806 31049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:22.934080 31049 log.cc:824] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:22.934974 31049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb: No bootstrap required, opened a new log
I20260807 08:42:22.937750 31049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7d7f7d04af4c81854181e0863f4fdb" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36229 } }
I20260807 08:42:22.937965 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 30964
I20260807 08:42:22.938042 31049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:22.938077 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:22.938092 31049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d7d7f7d04af4c81854181e0863f4fdb, State: Initialized, Role: FOLLOWER
I20260807 08:42:22.938269 29106 external_mini_cluster.cc:1428] Setting key 4cb3fb1db7bd87731f0e90c892e7faa6
I20260807 08:42:22.938294 31049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7d7f7d04af4c81854181e0863f4fdb" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36229 } }
I20260807 08:42:22.938405 31049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:22.938447 31049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:22.938571 31049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:22.939610 31049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7d7f7d04af4c81854181e0863f4fdb" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36229 } }
I20260807 08:42:22.939757 31049 leader_election.cc:302] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d7d7f7d04af4c81854181e0863f4fdb; no voters:
I20260807 08:42:22.939985 31049 leader_election.cc:288] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:22.940064 31054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:22.940277 31054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [term 1 LEADER]: Becoming Leader. State: Replica: 5d7d7f7d04af4c81854181e0863f4fdb, State: Running, Role: LEADER
I20260807 08:42:22.940325 31049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:22.940398 31054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7d7f7d04af4c81854181e0863f4fdb" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36229 } }
I20260807 08:42:22.940769 31054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d7d7f7d04af4c81854181e0863f4fdb. Latest consensus state: current_term: 1 leader_uuid: "5d7d7f7d04af4c81854181e0863f4fdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7d7f7d04af4c81854181e0863f4fdb" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36229 } } }
I20260807 08:42:22.940933 31054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:22.940778 31055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d7d7f7d04af4c81854181e0863f4fdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7d7f7d04af4c81854181e0863f4fdb" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36229 } } }
I20260807 08:42:22.941100 31055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:22.941227 31063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:22.941883 31063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:22.943760 31063 catalog_manager.cc:1347] Generated new cluster ID: 54e5038a3d4445b0a885f9ade56fc93d
I20260807 08:42:22.943830 31063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:42:22 dist-test-slave-9zdj krb5kdc[30948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:22.951396 31063 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:22.952255 31063 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:22.953135 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:22.938836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.28.108.190:36229)
0807 08:42:22.939050 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:22.939061 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:22.939127 (+ 66us) server_negotiation.cc:407] Connection header received
0807 08:42:22.939299 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:22.939305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:22.939355 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:22.939477 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:22.940481 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:22.942516 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:22.943387 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:22.944088 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:22.944215 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:22.947482 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:22.947488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:22.947488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:22.947505 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:22.950777 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:22.951495 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:22.951499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:22.951500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:22.951649 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:22.951907 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:22.951908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:22.951909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:22.952295 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:22.952524 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:22.952685 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 08:42:22.952864 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260807 08:42:22.958189 31063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb: Generated new TSK 0
I20260807 08:42:22.958441 31063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:42:22.993901 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:36229
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:23.094026 31076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:23.094297 31076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:23.094329 31076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:23.096303 31076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:23.096408 31076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:23.096439 31076 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 08:42:23.096453 31076 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 08:42:23.096498 31076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:23.099010 31076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.108.190:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:23.099409 31076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:23.099735 31076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:42:23.107116 31076 server_base.cc:1034] running on GCE node
W20260807 08:42:23.106942 31084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:23.106946 31081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:23.107018 31082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:23.107620 31076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:23.108006 31076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:23.109232 31076 hybrid_clock.cc:648] HybridClock initialized: now 1786092143109185 us; error 59 us; skew 500 ppm
Aug 07 08:42:23 dist-test-slave-9zdj krb5kdc[30948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092143, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:23.112664 31076 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:42:23.113207 31076 webserver.cc:460] Webserver started at http://127.28.108.129:39823/ using document root <none> and password file <none>
I20260807 08:42:23.113476 31076 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:23.113554 31076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:23.113776 31076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:23.115042 31076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "d9b437a4571b4dd1b24fca117406c1f6"
format_stamp: "Formatted at 2026-08-07 08:42:23 on dist-test-slave-9zdj"
server_key: "77fac94b28880efff8b13c1cb3f5b794"
server_key_iv: "b87e947ed4f2fc246442e9ed2e41b182"
server_key_version: "encryptionkey@0"
I20260807 08:42:23.115453 31076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "d9b437a4571b4dd1b24fca117406c1f6"
format_stamp: "Formatted at 2026-08-07 08:42:23 on dist-test-slave-9zdj"
server_key: "77fac94b28880efff8b13c1cb3f5b794"
server_key_iv: "b87e947ed4f2fc246442e9ed2e41b182"
server_key_version: "encryptionkey@0"
I20260807 08:42:23.117049 31076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260807 08:42:23.118096 31091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:23.118381 31076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:23.118512 31076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "d9b437a4571b4dd1b24fca117406c1f6"
format_stamp: "Formatted at 2026-08-07 08:42:23 on dist-test-slave-9zdj"
server_key: "77fac94b28880efff8b13c1cb3f5b794"
server_key_iv: "b87e947ed4f2fc246442e9ed2e41b182"
server_key_version: "encryptionkey@0"
I20260807 08:42:23.118604 31076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:23.130700 31076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:23.133749 31076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:23.133957 31076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:23.134568 31076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:23.134636 31076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:23.134671 31076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:23.134688 31076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 08:42:23 dist-test-slave-9zdj krb5kdc[30948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092143, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:23.142603 31076 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:44731
I20260807 08:42:23.142711 31222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:44731 every 8 connection(s)
I20260807 08:42:23.143044 31076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:42:23.146647 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.135409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:39093 (local address 127.28.108.190:36229)
0807 08:42:23.135480 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.135482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.135699 (+ 217us) server_negotiation.cc:407] Connection header received
0807 08:42:23.136527 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.136529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.136546 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.136630 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:23.137726 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.138629 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.139830 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.140212 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.140304 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.143221 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:23.143222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:23.143223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:23.143231 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:23.144975 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.145480 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.145482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.145482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.145557 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.145873 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.145874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.145875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.146014 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:23.146089 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.146414 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.146547 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 08:42:23.146858 31100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.135310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:36229 (local address 127.28.108.129:39093)
0807 08:42:23.135541 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.135595 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.136363 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.136753 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.136758 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.137241 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:23.137596 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.137603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.138772 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.138774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.139697 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.139700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.140352 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.140353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.140516 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.141250 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:23.141263 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:23.143100 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:23.145098 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.145108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.145110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.145387 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.145651 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.145652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.145652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.145791 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.146091 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:23.146100 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:23.146264 (+ 164us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.146374 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.146562 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":97,"threads_started":1}
I20260807 08:42:23.147506 31223 heartbeater.cc:347] Connected to a master server at 127.28.108.190:36229
I20260807 08:42:23.147608 31223 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:23.147830 31223 heartbeater.cc:511] Master 127.28.108.190:36229 requested a full tablet report, sending...
I20260807 08:42:23.148627 30998 ts_manager.cc:194] Registered new tserver with Master: d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731)
I20260807 08:42:23.149696 30998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:39093
I20260807 08:42:23.151530 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 31076
I20260807 08:42:23.151654 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:23.151979 29106 external_mini_cluster.cc:1428] Setting key 5dd0e36102a224d5d29b163699df9dbe
I20260807 08:42:23.156054 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.148873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:48355 (local address 127.28.108.190:36229)
0807 08:42:23.148928 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.148931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.149068 (+ 137us) server_negotiation.cc:407] Connection header received
0807 08:42:23.149116 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.149117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.149126 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.149176 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:23.149552 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.150253 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.151075 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.151281 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.151345 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.152585 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:23.152587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:23.152587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:23.152593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:23.154533 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.155109 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.155110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.155110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.155180 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.155469 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.155470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.155471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.155604 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:23.155711 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.155885 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.155970 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 08:42:23.156131 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.148828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:36229 (local address 127.28.108.129:48355)
0807 08:42:23.148986 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.148998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.149064 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.149182 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.149183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.149398 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:23.149464 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.149466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.150357 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.150358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.150989 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.150991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.151366 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.151367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.151454 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.152001 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:23.152008 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:23.152467 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:23.154677 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.154678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.154679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.154997 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.155262 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.155263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.155263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.155351 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.155707 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:23.155708 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:23.155775 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.155858 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.155988 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:42:23.189980 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:36229
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:23.286233 31232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:23.286478 31232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:23.286509 31232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:23.287782 31232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:23.287837 31232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:23.287861 31232 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 08:42:23.287873 31232 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 08:42:23.287923 31232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:23.289566 31232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.108.190:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:23.289855 31232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:23.290144 31232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:23.296422 31243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:23.296756 31239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:23.296725 31241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:23.298133 31232 server_base.cc:1034] running on GCE node
I20260807 08:42:23.298640 31232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:23.298980 31232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:23.300182 31232 hybrid_clock.cc:648] HybridClock initialized: now 1786092143300149 us; error 54 us; skew 500 ppm
Aug 07 08:42:23 dist-test-slave-9zdj krb5kdc[30948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092143, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:23.303561 31232 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:42:23.304195 31232 webserver.cc:460] Webserver started at http://127.28.108.130:41249/ using document root <none> and password file <none>
I20260807 08:42:23.304451 31232 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:23.304527 31232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:23.304690 31232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:23.305934 31232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "d22fd5c074e24a4f98ae251be8d8c7f5"
format_stamp: "Formatted at 2026-08-07 08:42:23 on dist-test-slave-9zdj"
server_key: "a858630cacb0baf1aa4593906983deda"
server_key_iv: "2f91fb9427f9942a604ea699a70a6fbd"
server_key_version: "encryptionkey@0"
I20260807 08:42:23.306324 31232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "d22fd5c074e24a4f98ae251be8d8c7f5"
format_stamp: "Formatted at 2026-08-07 08:42:23 on dist-test-slave-9zdj"
server_key: "a858630cacb0baf1aa4593906983deda"
server_key_iv: "2f91fb9427f9942a604ea699a70a6fbd"
server_key_version: "encryptionkey@0"
I20260807 08:42:23.307963 31232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:23.309069 31251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:23.309484 31232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:23.309659 31232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "d22fd5c074e24a4f98ae251be8d8c7f5"
format_stamp: "Formatted at 2026-08-07 08:42:23 on dist-test-slave-9zdj"
server_key: "a858630cacb0baf1aa4593906983deda"
server_key_iv: "2f91fb9427f9942a604ea699a70a6fbd"
server_key_version: "encryptionkey@0"
I20260807 08:42:23.309777 31232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:23.341555 31232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:23.344508 31232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:23.344733 31232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:23.345391 31232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:23.345468 31232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:23.345525 31232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:23.345557 31232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 08:42:23 dist-test-slave-9zdj krb5kdc[30948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092143, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:23.354352 31232 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:41993
I20260807 08:42:23.354777 31232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:23.354748 31386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:41993 every 8 connection(s)
I20260807 08:42:23.357549 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.346006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:48411 (local address 127.28.108.190:36229)
0807 08:42:23.346092 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.346094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.346323 (+ 229us) server_negotiation.cc:407] Connection header received
0807 08:42:23.346936 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.346937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.346952 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.347036 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:23.348050 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.348591 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.349947 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.350223 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.350314 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.354033 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:23.354035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:23.354035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:23.354045 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:23.355942 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.356368 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.356369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.356370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.356430 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.356822 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.356823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.356824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.356957 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:23.357017 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.357372 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.357480 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 08:42:23.357844 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.345964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:36229 (local address 127.28.108.130:48411)
0807 08:42:23.346159 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.346205 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.346809 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.347126 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.347130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.347692 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:23.347941 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.347946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.348726 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.348727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.349847 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.349850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.350336 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.350337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.350479 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.351164 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:23.351179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:23.353905 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:23.356038 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.356044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.356046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.356286 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.356495 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.356495 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.356495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.356603 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.357041 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:23.357048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:23.357226 (+ 178us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.357341 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.357495 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":119,"threads_started":1}
I20260807 08:42:23.358026 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 31232
I20260807 08:42:23.358140 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:42:23.358348 29106 external_mini_cluster.cc:1428] Setting key 82724926869a90db806fb9ba43a9f4f0
I20260807 08:42:23.358615 31387 heartbeater.cc:347] Connected to a master server at 127.28.108.190:36229
I20260807 08:42:23.358721 31387 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:23.358927 31387 heartbeater.cc:511] Master 127.28.108.190:36229 requested a full tablet report, sending...
I20260807 08:42:23.359308 30998 ts_manager.cc:194] Registered new tserver with Master: d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130:41993)
I20260807 08:42:23.359833 30998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:48411
I20260807 08:42:23.368093 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.359776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:36229 (local address 127.28.108.130:33389)
0807 08:42:23.359947 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.359962 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.360020 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.361347 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.361349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.361666 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:23.361751 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.361753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.362568 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.362569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.363204 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.363206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.363775 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.363777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.363851 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.364489 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:23.364493 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:23.364829 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:23.366911 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.366912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.366912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.367122 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.367355 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.367356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.367356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.367419 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.367796 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:23.367797 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:23.367847 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.367903 (+ 56us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.368008 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260807 08:42:23.368176 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.361116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:33389 (local address 127.28.108.190:36229)
0807 08:42:23.361188 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.361190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.361200 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:23.361223 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.361224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.361235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.361299 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:23.361850 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.362433 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.363301 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.363634 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.363723 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.364932 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:23.364934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:23.364934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:23.364944 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:23.366796 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.367205 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.367206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.367207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.367281 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.367485 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.367486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.367487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.367725 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:23.367805 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.367956 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.368049 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:42:23.395150 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:36229
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:23.491453 31398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:23.491758 31398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:23.491873 31398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:23.493351 31398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:23.493444 31398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:23.493484 31398 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 08:42:23.493510 31398 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 08:42:23.493548 31398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:23.495275 31398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.108.190:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:23.495617 31398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:23.495909 31398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:23.502013 31403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:23.501995 31408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:23.502220 31398 server_base.cc:1034] running on GCE node
W20260807 08:42:23.501986 31404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:23.502620 31398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:23.502882 31398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:23.504180 31398 hybrid_clock.cc:648] HybridClock initialized: now 1786092143504162 us; error 30 us; skew 500 ppm
Aug 07 08:42:23 dist-test-slave-9zdj krb5kdc[30948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092143, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:23.507706 31398 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:42:23.508353 31398 webserver.cc:460] Webserver started at http://127.28.108.131:33517/ using document root <none> and password file <none>
I20260807 08:42:23.508625 31398 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:23.508713 31398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:23.508893 31398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:23.510193 31398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "4464d44169ee486d8cc524d0aac36422"
format_stamp: "Formatted at 2026-08-07 08:42:23 on dist-test-slave-9zdj"
server_key: "3e65acc6915967d6a89acde99ba3dc75"
server_key_iv: "9d9f062be2a333e8ba80e46fcdf45b52"
server_key_version: "encryptionkey@0"
I20260807 08:42:23.510656 31398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "4464d44169ee486d8cc524d0aac36422"
format_stamp: "Formatted at 2026-08-07 08:42:23 on dist-test-slave-9zdj"
server_key: "3e65acc6915967d6a89acde99ba3dc75"
server_key_iv: "9d9f062be2a333e8ba80e46fcdf45b52"
server_key_version: "encryptionkey@0"
I20260807 08:42:23.512466 31398 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 08:42:23.513700 31416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:23.513960 31398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:23.514101 31398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "4464d44169ee486d8cc524d0aac36422"
format_stamp: "Formatted at 2026-08-07 08:42:23 on dist-test-slave-9zdj"
server_key: "3e65acc6915967d6a89acde99ba3dc75"
server_key_iv: "9d9f062be2a333e8ba80e46fcdf45b52"
server_key_version: "encryptionkey@0"
I20260807 08:42:23.514286 31398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:23.523459 31398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:23.526908 31398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:23.527158 31398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:23.527859 31398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:23.527943 31398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:23.527995 31398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:23.528044 31398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:23.536842 31398 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:43827
I20260807 08:42:23.536958 31540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:43827 every 8 connection(s)
Aug 07 08:42:23 dist-test-slave-9zdj krb5kdc[30948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092143, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:23.537381 31398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:23.541510 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.528564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:47803 (local address 127.28.108.190:36229)
0807 08:42:23.528657 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.528659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.528862 (+ 203us) server_negotiation.cc:407] Connection header received
0807 08:42:23.529453 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.529455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.529471 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.529568 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:23.530806 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.531786 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.533664 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.534102 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.534214 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.537910 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:23.537912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:23.537913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:23.537923 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:23.539597 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.540104 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.540106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.540106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.540171 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.540511 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.540513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.540513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.540663 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:23.540739 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.541335 (+ 596us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.541433 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 08:42:23.541769 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.528486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:36229 (local address 127.28.108.131:47803)
0807 08:42:23.528705 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.528757 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.529309 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.529724 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.529729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.530133 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:23.530641 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.530648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.531940 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.531942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.533511 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.533515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.534252 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.534253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.534446 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.535250 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:23.535265 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:23.537738 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:23.539715 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.539724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.539726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.540028 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.540258 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.540259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.540259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.540399 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.540855 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:23.540864 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:23.541072 (+ 208us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.541290 (+ 218us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.541463 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":118,"threads_started":1}
I20260807 08:42:23.542421 31541 heartbeater.cc:347] Connected to a master server at 127.28.108.190:36229
I20260807 08:42:23.542524 31541 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:23.542723 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 31398
I20260807 08:42:23.542831 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:23.542840 31541 heartbeater.cc:511] Master 127.28.108.190:36229 requested a full tablet report, sending...
I20260807 08:42:23.543092 29106 external_mini_cluster.cc:1428] Setting key 144f86ecbb734dfc82b0e7c3b189f65f
I20260807 08:42:23.543301 30997 ts_manager.cc:194] Registered new tserver with Master: 4464d44169ee486d8cc524d0aac36422 (127.28.108.131:43827)
I20260807 08:42:23.543856 30997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:47803
I20260807 08:42:23.543958 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:42:23.551587 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.543797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:41247 (local address 127.28.108.190:36229)
0807 08:42:23.543907 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.543909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.544040 (+ 131us) server_negotiation.cc:407] Connection header received
0807 08:42:23.544133 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.544134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.544144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.544201 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:23.544629 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.545204 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.546111 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.546476 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.546563 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.547959 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:23.547961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:23.547961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:23.547971 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:23.550013 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.550441 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.550442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.550443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.550523 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.550910 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.550911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.550912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.551109 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:23.551174 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.551418 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.551502 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 08:42:23.551931 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.543762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:36229 (local address 127.28.108.131:41247)
0807 08:42:23.543977 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.543990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.544067 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.544219 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.544220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.544471 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:23.544537 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.544539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.545324 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.545325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.546013 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.546015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.546620 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.546621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.546703 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.547467 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:23.547472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:23.547846 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:23.550131 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.550133 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.550133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.550336 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.550640 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.550641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.550641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.550783 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.551202 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:23.551203 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:23.551279 (+ 76us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.551383 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.551600 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
Aug 07 08:42:23 dist-test-slave-9zdj krb5kdc[30948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 08:42:23 dist-test-slave-9zdj krb5kdc[30948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:23.574847 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.566429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36268 (local address 127.28.108.190:36229)
0807 08:42:23.566515 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.566518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.566534 (+ 16us) server_negotiation.cc:407] Connection header received
0807 08:42:23.566561 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.566562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.566571 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.566633 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:23.567325 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.567847 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.568663 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.568869 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.568950 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.571800 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:23.571802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:23.571802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:23.571811 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:23.573346 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.573913 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.573914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.573915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.573989 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.574236 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.574237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.574237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.574380 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:23.574502 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.574617 (+ 115us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.574727 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 08:42:23.576704 30997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:23.578581 30997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:23.586419 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.581520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:43827 (local address 127.0.0.1:58656)
0807 08:42:23.581738 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.581782 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.581875 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.582552 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.582553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.582555 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:23.582623 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.582625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.584008 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.584008 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.584868 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.584871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.586014 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.586015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.586121 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.586138 (+ 17us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.586221 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.586292 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":130,"threads_started":1}
I20260807 08:42:23.586447 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.581791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.28.108.131:43827)
0807 08:42:23.582265 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.582270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.582284 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:42:23.582435 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.582437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.582458 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.582553 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:23.582699 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.583888 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.584981 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.585826 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.585907 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.585971 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.586260 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.586313 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":114,"threads_started":1}
I20260807 08:42:23.586422 31563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.581568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:44731 (local address 127.0.0.1:38568)
0807 08:42:23.582030 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.582037 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.582070 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.582463 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.582466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.582469 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:23.582558 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.582561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.583713 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.583714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.585106 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.585109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.586200 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.586201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.586273 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.586277 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.586324 (+ 47us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.586362 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":24,"thread_start_us":433,"threads_started":1}
I20260807 08:42:23.586756 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.582002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.28.108.129:44731)
0807 08:42:23.582204 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.582210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.582222 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:42:23.582268 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.582270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.582288 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.582380 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:23.582650 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.583603 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.585231 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.586037 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.586108 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.586164 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.586467 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.586517 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":111,"threads_started":1}
I20260807 08:42:23.587181 31560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.581520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41993 (local address 127.0.0.1:54918)
0807 08:42:23.581822 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.581832 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.581876 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.582877 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.582878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.582880 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:23.582919 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.582920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.584134 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.584135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.585344 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.585346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.586900 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.586901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.586984 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.586989 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.587045 (+ 56us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.587099 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260807 08:42:23.587589 31147 tablet_service.cc:1467] Processing CreateTablet for tablet ea3ec0e1c7c94315ae8999be84f06f65 (DEFAULT_TABLE table=test-table [id=32d63318fa02417c85af887ac736f08d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:23.587602 31469 tablet_service.cc:1467] Processing CreateTablet for tablet ea3ec0e1c7c94315ae8999be84f06f65 (DEFAULT_TABLE table=test-table [id=32d63318fa02417c85af887ac736f08d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:23.587895 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.581767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.28.108.130:41993)
0807 08:42:23.582701 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.582705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.582716 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:42:23.582755 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.582758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.582774 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.582858 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:23.582988 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.584042 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.585437 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.586498 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.587606 (+ 1108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.587691 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.587721 (+ 30us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.587772 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":87,"threads_started":1}
I20260807 08:42:23.588040 31147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3ec0e1c7c94315ae8999be84f06f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:23.588052 31469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3ec0e1c7c94315ae8999be84f06f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:23.588627 31310 tablet_service.cc:1467] Processing CreateTablet for tablet ea3ec0e1c7c94315ae8999be84f06f65 (DEFAULT_TABLE table=test-table [id=32d63318fa02417c85af887ac736f08d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:23.589025 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3ec0e1c7c94315ae8999be84f06f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:23.591225 31565 tablet_bootstrap.cc:492] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422: Bootstrap starting.
I20260807 08:42:23.592244 31566 tablet_bootstrap.cc:492] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: Bootstrap starting.
I20260807 08:42:23.592981 31565 tablet_bootstrap.cc:654] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:23.592981 31566 tablet_bootstrap.cc:654] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:23.593024 31567 tablet_bootstrap.cc:492] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: Bootstrap starting.
I20260807 08:42:23.593431 31566 log.cc:824] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:23.593559 31565 log.cc:824] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:23.593788 31567 tablet_bootstrap.cc:654] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:23.594165 31567 log.cc:824] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:23.594456 31566 tablet_bootstrap.cc:492] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: No bootstrap required, opened a new log
I20260807 08:42:23.594488 31565 tablet_bootstrap.cc:492] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422: No bootstrap required, opened a new log
I20260807 08:42:23.594563 31565 ts_tablet_manager.cc:1397] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:42:23.594534 31566 ts_tablet_manager.cc:1397] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:42:23.595405 31567 tablet_bootstrap.cc:492] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: No bootstrap required, opened a new log
I20260807 08:42:23.595525 31567 ts_tablet_manager.cc:1397] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260807 08:42:23.596747 31565 raft_consensus.cc:348] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } }
I20260807 08:42:23.596873 31566 raft_consensus.cc:348] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } }
I20260807 08:42:23.597007 31565 raft_consensus.cc:374] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:23.597045 31566 raft_consensus.cc:374] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:23.597065 31565 raft_consensus.cc:729] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4464d44169ee486d8cc524d0aac36422, State: Initialized, Role: FOLLOWER
I20260807 08:42:23.597085 31566 raft_consensus.cc:729] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b437a4571b4dd1b24fca117406c1f6, State: Initialized, Role: FOLLOWER
I20260807 08:42:23.597235 31566 consensus_queue.cc:260] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } }
I20260807 08:42:23.597234 31565 consensus_queue.cc:260] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } }
I20260807 08:42:23.597555 31565 ts_tablet_manager.cc:1428] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:42:23.597556 31566 ts_tablet_manager.cc:1428] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:42:23.597751 31223 heartbeater.cc:503] Master 127.28.108.190:36229 was elected leader, sending a full tablet report...
I20260807 08:42:23.597771 31541 heartbeater.cc:503] Master 127.28.108.190:36229 was elected leader, sending a full tablet report...
I20260807 08:42:23.597955 31567 raft_consensus.cc:348] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } }
I20260807 08:42:23.598127 31567 raft_consensus.cc:374] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:23.598165 31567 raft_consensus.cc:729] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22fd5c074e24a4f98ae251be8d8c7f5, State: Initialized, Role: FOLLOWER
I20260807 08:42:23.598287 31567 consensus_queue.cc:260] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } }
I20260807 08:42:23.598598 31567 ts_tablet_manager.cc:1428] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 08:42:23.598613 31387 heartbeater.cc:503] Master 127.28.108.190:36229 was elected leader, sending a full tablet report...
W20260807 08:42:23.605685 31389 tablet.cc:2369] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:42:23.644001 31224 tablet.cc:2369] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:23.681525 31574 raft_consensus.cc:482] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:23.681761 31574 raft_consensus.cc:504] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } }
I20260807 08:42:23.682267 31574 leader_election.cc:288] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4464d44169ee486d8cc524d0aac36422 (127.28.108.131:43827), d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731)
I20260807 08:42:23.685849 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.682528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:44731 (local address 127.28.108.130:44249)
0807 08:42:23.682752 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.682763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.682870 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.683039 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.683040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.683042 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:23.683146 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.683147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.683895 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.683896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.684782 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.684784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.685471 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.685473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.685565 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.685569 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.685678 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.685728 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":19,"thread_start_us":116,"threads_started":1}
I20260807 08:42:23.685818 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.682601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:44249 (local address 127.28.108.129:44731)
0807 08:42:23.682659 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.682661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.682854 (+ 193us) server_negotiation.cc:407] Connection header received
0807 08:42:23.682945 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.682946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.682963 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.683036 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:23.683226 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.683788 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.684906 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.685362 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.685459 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.685469 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.685692 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.685743 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 08:42:23.685948 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.682528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:43827 (local address 127.28.108.130:53775)
0807 08:42:23.682596 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.682610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.682765 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.682931 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.682931 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.682933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:23.683027 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.683029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.684029 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.684031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.684905 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.684908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.685600 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.685601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.685729 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.685732 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.685815 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.685865 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 08:42:23.685957 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.682686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:53775 (local address 127.28.108.131:43827)
0807 08:42:23.682751 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.682754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.682766 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:42:23.682832 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.682833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.682844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.682923 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:23.683114 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.683897 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.685023 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.685493 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.685602 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.685611 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.685844 (+ 233us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.685891 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 08:42:23.686414 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3ec0e1c7c94315ae8999be84f06f65" candidate_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4464d44169ee486d8cc524d0aac36422" is_pre_election: true
I20260807 08:42:23.686493 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3ec0e1c7c94315ae8999be84f06f65" candidate_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b437a4571b4dd1b24fca117406c1f6" is_pre_election: true
I20260807 08:42:23.686625 31490 raft_consensus.cc:2454] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d22fd5c074e24a4f98ae251be8d8c7f5 in term 0.
I20260807 08:42:23.686692 31168 raft_consensus.cc:2454] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d22fd5c074e24a4f98ae251be8d8c7f5 in term 0.
I20260807 08:42:23.686954 31254 leader_election.cc:302] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4464d44169ee486d8cc524d0aac36422, d22fd5c074e24a4f98ae251be8d8c7f5; no voters:
I20260807 08:42:23.687125 31574 raft_consensus.cc:2781] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:23.687193 31574 raft_consensus.cc:482] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:23.687232 31574 raft_consensus.cc:3037] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:23.688364 31574 raft_consensus.cc:504] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } }
I20260807 08:42:23.688562 31574 leader_election.cc:288] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [CANDIDATE]: Term 1 election: Requested vote from peers 4464d44169ee486d8cc524d0aac36422 (127.28.108.131:43827), d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731)
I20260807 08:42:23.688781 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3ec0e1c7c94315ae8999be84f06f65" candidate_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4464d44169ee486d8cc524d0aac36422"
I20260807 08:42:23.688890 31490 raft_consensus.cc:3037] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:23.688970 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3ec0e1c7c94315ae8999be84f06f65" candidate_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b437a4571b4dd1b24fca117406c1f6"
I20260807 08:42:23.689074 31168 raft_consensus.cc:3037] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:23.690152 31168 raft_consensus.cc:2454] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d22fd5c074e24a4f98ae251be8d8c7f5 in term 1.
I20260807 08:42:23.690164 31490 raft_consensus.cc:2454] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d22fd5c074e24a4f98ae251be8d8c7f5 in term 1.
I20260807 08:42:23.690542 31253 leader_election.cc:302] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d22fd5c074e24a4f98ae251be8d8c7f5, d9b437a4571b4dd1b24fca117406c1f6; no voters:
I20260807 08:42:23.690687 31574 raft_consensus.cc:2781] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:23.690881 31574 raft_consensus.cc:686] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 LEADER]: Becoming Leader. State: Replica: d22fd5c074e24a4f98ae251be8d8c7f5, State: Running, Role: LEADER
I20260807 08:42:23.691020 31574 consensus_queue.cc:237] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } }
I20260807 08:42:23.692211 30997 catalog_manager.cc:5495] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130). New cstate: current_term: 1 leader_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } health_report { overall_health: HEALTHY } } }
Aug 07 08:42:23 dist-test-slave-9zdj krb5kdc[30948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:23.735975 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.726516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:36229 (local address 127.0.0.1:36274)
0807 08:42:23.726739 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.726758 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.726833 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.726972 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.726973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.727335 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:23.727402 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.727405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.728252 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.728253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.728900 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.728904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.729470 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.729471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.729537 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.730275 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:23.730280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:23.732125 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:23.734425 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.734439 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.734442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.734740 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.735047 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:23.735048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:23.735048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:23.735133 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:23.735546 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:23.735557 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:23.735630 (+ 73us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.735721 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.735863 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260807 08:42:23.736079 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.726637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.28.108.190:36229)
0807 08:42:23.726830 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.726832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.726844 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:42:23.726877 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.726878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.726888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.726945 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:23.727517 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.728096 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.729027 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.729337 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.729411 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.732246 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:23.732248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:23.732248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:23.732256 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:23.734291 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.734877 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.734878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.734878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.734949 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:23.735201 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:23.735202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:23.735202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:23.735440 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:23.735505 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.735770 (+ 265us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.735978 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":150}
I20260807 08:42:23.739456 30997 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:36274:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 08:42:23.739763 30997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:23.741881 31310 tablet_service.cc:1467] Processing CreateTablet for tablet b7fb7039169241648081a0b6ce407d83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ea37ebd3e394b44bfad8730fffe63e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 08:42:23.741907 31469 tablet_service.cc:1467] Processing CreateTablet for tablet b7fb7039169241648081a0b6ce407d83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ea37ebd3e394b44bfad8730fffe63e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 08:42:23.741890 31147 tablet_service.cc:1467] Processing CreateTablet for tablet b7fb7039169241648081a0b6ce407d83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ea37ebd3e394b44bfad8730fffe63e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 08:42:23.742093 31469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fb7039169241648081a0b6ce407d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:23.742093 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fb7039169241648081a0b6ce407d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:23.742122 31147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fb7039169241648081a0b6ce407d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:23.743602 31566 tablet_bootstrap.cc:492] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6: Bootstrap starting.
I20260807 08:42:23.743741 31565 tablet_bootstrap.cc:492] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: Bootstrap starting.
I20260807 08:42:23.743752 31567 tablet_bootstrap.cc:492] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: Bootstrap starting.
I20260807 08:42:23.744114 31566 tablet_bootstrap.cc:654] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:23.744223 31567 tablet_bootstrap.cc:654] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:23.744400 31565 tablet_bootstrap.cc:654] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:23.745044 31567 tablet_bootstrap.cc:492] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: No bootstrap required, opened a new log
I20260807 08:42:23.745090 31566 tablet_bootstrap.cc:492] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6: No bootstrap required, opened a new log
I20260807 08:42:23.745101 31567 ts_tablet_manager.cc:1397] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:23.745119 31565 tablet_bootstrap.cc:492] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: No bootstrap required, opened a new log
I20260807 08:42:23.745151 31566 ts_tablet_manager.cc:1397] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 08:42:23.745182 31565 ts_tablet_manager.cc:1397] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:23.745309 31565 raft_consensus.cc:348] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.745311 31566 raft_consensus.cc:348] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.745373 31565 raft_consensus.cc:374] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:23.745386 31565 raft_consensus.cc:729] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4464d44169ee486d8cc524d0aac36422, State: Initialized, Role: FOLLOWER
I20260807 08:42:23.745304 31567 raft_consensus.cc:348] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.745383 31566 raft_consensus.cc:374] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:23.745407 31567 raft_consensus.cc:374] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:23.745409 31566 raft_consensus.cc:729] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b437a4571b4dd1b24fca117406c1f6, State: Initialized, Role: FOLLOWER
I20260807 08:42:23.745430 31567 raft_consensus.cc:729] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22fd5c074e24a4f98ae251be8d8c7f5, State: Initialized, Role: FOLLOWER
I20260807 08:42:23.745455 31565 consensus_queue.cc:260] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.745471 31566 consensus_queue.cc:260] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.745479 31567 consensus_queue.cc:260] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.745553 31565 ts_tablet_manager.cc:1428] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 08:42:23.745565 31566 ts_tablet_manager.cc:1428] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:23.745565 31567 ts_tablet_manager.cc:1428] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 08:42:23.745649 31573 tablet_replica.cc:439] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } }
I20260807 08:42:23.745733 31573 tablet_replica.cc:442] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:23.745733 31572 tablet_replica.cc:439] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } }
I20260807 08:42:23.745815 31572 tablet_replica.cc:442] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:23.745884 31574 tablet_replica.cc:439] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } }
I20260807 08:42:23.745997 31574 tablet_replica.cc:442] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 08:42:23.788285 31542 tablet.cc:2369] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:23.836112 31574 raft_consensus.cc:482] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:23.836259 31574 raft_consensus.cc:504] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.836503 31574 leader_election.cc:288] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731), 4464d44169ee486d8cc524d0aac36422 (127.28.108.131:43827)
I20260807 08:42:23.836768 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fb7039169241648081a0b6ce407d83" candidate_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b437a4571b4dd1b24fca117406c1f6" is_pre_election: true
I20260807 08:42:23.836887 31168 raft_consensus.cc:2454] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d22fd5c074e24a4f98ae251be8d8c7f5 in term 0.
I20260807 08:42:23.836959 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fb7039169241648081a0b6ce407d83" candidate_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4464d44169ee486d8cc524d0aac36422" is_pre_election: true
I20260807 08:42:23.837118 31490 raft_consensus.cc:2454] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d22fd5c074e24a4f98ae251be8d8c7f5 in term 0.
I20260807 08:42:23.837091 31253 leader_election.cc:302] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d22fd5c074e24a4f98ae251be8d8c7f5, d9b437a4571b4dd1b24fca117406c1f6; no voters:
I20260807 08:42:23.837363 31574 raft_consensus.cc:2781] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:23.837442 31574 raft_consensus.cc:482] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:23.837471 31574 raft_consensus.cc:3037] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:23.838284 31574 raft_consensus.cc:504] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.838479 31574 leader_election.cc:288] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [CANDIDATE]: Term 1 election: Requested vote from peers d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731), 4464d44169ee486d8cc524d0aac36422 (127.28.108.131:43827)
I20260807 08:42:23.838712 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fb7039169241648081a0b6ce407d83" candidate_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b437a4571b4dd1b24fca117406c1f6"
I20260807 08:42:23.838832 31168 raft_consensus.cc:3037] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:23.838855 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fb7039169241648081a0b6ce407d83" candidate_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4464d44169ee486d8cc524d0aac36422"
I20260807 08:42:23.838943 31490 raft_consensus.cc:3037] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:23.839843 31490 raft_consensus.cc:2454] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d22fd5c074e24a4f98ae251be8d8c7f5 in term 1.
I20260807 08:42:23.839896 31168 raft_consensus.cc:2454] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d22fd5c074e24a4f98ae251be8d8c7f5 in term 1.
I20260807 08:42:23.840071 31253 leader_election.cc:302] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d22fd5c074e24a4f98ae251be8d8c7f5, d9b437a4571b4dd1b24fca117406c1f6; no voters:
I20260807 08:42:23.840320 31574 raft_consensus.cc:2781] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:23.840400 31574 raft_consensus.cc:686] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 LEADER]: Becoming Leader. State: Replica: d22fd5c074e24a4f98ae251be8d8c7f5, State: Running, Role: LEADER
I20260807 08:42:23.840453 31574 consensus_queue.cc:237] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.840854 31576 tablet_replica.cc:439] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: TxnStatusTablet state changed. Reason: New leader d22fd5c074e24a4f98ae251be8d8c7f5. Latest consensus state: current_term: 1 leader_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } }
I20260807 08:42:23.840961 31576 tablet_replica.cc:442] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: This TxnStatusTablet replica's current role is: LEADER
I20260807 08:42:23.841194 31588 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 08:42:23.841214 30998 catalog_manager.cc:5495] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130). New cstate: current_term: 1 leader_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } health_report { overall_health: UNKNOWN } } }
I20260807 08:42:23.841380 31588 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 08:42:23.868535 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.865535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41993 (local address 127.0.0.1:54928)
0807 08:42:23.865656 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.865679 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.865766 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.866010 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.866011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.866013 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:23.866077 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.866080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.866831 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.866833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.867451 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.867453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.867866 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.867867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.867945 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.867977 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:23.868318 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:23.868332 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.868383 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.868463 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260807 08:42:23.868518 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.865720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.28.108.130:41993)
0807 08:42:23.865795 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.865797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.865807 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:23.865887 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.865887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.865910 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.866000 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:23.866157 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.866668 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.867555 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.867783 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.867865 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.868098 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:23.868222 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:23.868324 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.868398 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.868429 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 08:42:23.870879 31490 raft_consensus.cc:1261] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Refusing update from remote peer d22fd5c074e24a4f98ae251be8d8c7f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:23.870949 31168 raft_consensus.cc:1261] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Refusing update from remote peer d22fd5c074e24a4f98ae251be8d8c7f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:23.871276 31576 consensus_queue.cc:1035] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:23.871464 31574 consensus_queue.cc:1035] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:23.872852 31572 tablet_replica.cc:439] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6: TxnStatusTablet state changed. Reason: New leader d22fd5c074e24a4f98ae251be8d8c7f5. Latest consensus state: current_term: 1 leader_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } }
I20260807 08:42:23.872957 31572 tablet_replica.cc:442] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:23.873095 31573 tablet_replica.cc:439] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: TxnStatusTablet state changed. Reason: New leader d22fd5c074e24a4f98ae251be8d8c7f5. Latest consensus state: current_term: 1 leader_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } }
I20260807 08:42:23.873217 31573 tablet_replica.cc:442] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:23.874473 31572 tablet_replica.cc:439] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } }
I20260807 08:42:23.874475 31573 tablet_replica.cc:439] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } }
I20260807 08:42:23.874496 31574 tablet_replica.cc:439] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } }
I20260807 08:42:23.874567 31572 tablet_replica.cc:442] T b7fb7039169241648081a0b6ce407d83 P d9b437a4571b4dd1b24fca117406c1f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:23.874567 31573 tablet_replica.cc:442] T b7fb7039169241648081a0b6ce407d83 P 4464d44169ee486d8cc524d0aac36422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:23.874574 31574 tablet_replica.cc:442] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: This TxnStatusTablet replica's current role is: LEADER
I20260807 08:42:23.874622 31574 tablet_replica.cc:439] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } } }
I20260807 08:42:23.874657 31574 tablet_replica.cc:442] T b7fb7039169241648081a0b6ce407d83 P d22fd5c074e24a4f98ae251be8d8c7f5: This TxnStatusTablet replica's current role is: LEADER
I20260807 08:42:23.875138 31592 mvcc.cc:205] Tried to move back new op lower bound from 7315833421292290048 to 7315833421171191808. Current Snapshot: MvccSnapshot[applied={T|T < 7315833421292290048}]
I20260807 08:42:23.875634 31594 mvcc.cc:205] Tried to move back new op lower bound from 7315833421292290048 to 7315833421171191808. Current Snapshot: MvccSnapshot[applied={T|T < 7315833421292290048}]
I20260807 08:42:23.876259 31593 mvcc.cc:205] Tried to move back new op lower bound from 7315833421292290048 to 7315833421171191808. Current Snapshot: MvccSnapshot[applied={T|T < 7315833421292290048}]
I20260807 08:42:23.880261 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.877238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.28.108.130:41993)
0807 08:42:23.877409 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.877410 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.877417 (+ 7us) server_negotiation.cc:407] Connection header received
0807 08:42:23.877435 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.877436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.877444 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.877516 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:23.877698 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.878476 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.879394 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.879708 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.879793 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.879960 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:23.880000 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:23.880056 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.880141 (+ 85us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.880172 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260807 08:42:23.889079 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.884960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:45081 (local address 127.28.108.130:41993)
0807 08:42:23.885076 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:23.885079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:23.885727 (+ 648us) server_negotiation.cc:407] Connection header received
0807 08:42:23.885757 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:23.885758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:23.885770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:23.885835 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:23.886199 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.886956 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.888151 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.888458 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.888541 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.888795 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:23.888831 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:23.888894 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:23.888983 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 08:42:23.889003 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 08:42:23.889179 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:23.884882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41993 (local address 127.28.108.130:45081)
0807 08:42:23.885341 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:23.885354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:23.885651 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:23.885862 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:23.885863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:23.886032 (+ 169us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:23.886113 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.886115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.887046 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.887047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.888044 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:23.888046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:23.888552 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:23.888553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:23.888626 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:23.888648 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:23.888916 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:23.888918 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 08:42:23.888969 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 08:42:23.889018 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":340,"threads_started":1}
I20260807 08:42:23.892758 31490 raft_consensus.cc:1261] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Refusing update from remote peer d22fd5c074e24a4f98ae251be8d8c7f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:23.892954 31168 raft_consensus.cc:1261] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Refusing update from remote peer d22fd5c074e24a4f98ae251be8d8c7f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:23.892992 31598 consensus_queue.cc:1035] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:23.893325 31598 consensus_queue.cc:1035] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:23.908552 30998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:23.908705 30998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:23.909588 30998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb: Sending DeleteTablet for 3 replicas of tablet ea3ec0e1c7c94315ae8999be84f06f65
I20260807 08:42:23.910046 31469 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3ec0e1c7c94315ae8999be84f06f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:23 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:58656
I20260807 08:42:23.910096 31147 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3ec0e1c7c94315ae8999be84f06f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:23 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:38568
I20260807 08:42:23.910189 31310 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3ec0e1c7c94315ae8999be84f06f65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:23 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:54918
I20260807 08:42:23.910516 31612 tablet_replica.cc:330] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422: stopping tablet replica
I20260807 08:42:23.910580 30998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 08:42:23.910660 31612 raft_consensus.cc:2229] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:23.910725 31612 raft_consensus.cc:2258] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 08:42:23.910760 30998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:23.911233 31612 ts_tablet_manager.cc:1905] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:23.912267 31147 tablet_service.cc:1467] Processing CreateTablet for tablet c9d0511b167344ecae8a20ddf4c5a305 (DEFAULT_TABLE table=test-table [id=d929ea0d50a74ea492e05dab6775ee64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:23.912307 31469 tablet_service.cc:1467] Processing CreateTablet for tablet c9d0511b167344ecae8a20ddf4c5a305 (DEFAULT_TABLE table=test-table [id=d929ea0d50a74ea492e05dab6775ee64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:23.912412 31147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d0511b167344ecae8a20ddf4c5a305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:23.912444 31469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d0511b167344ecae8a20ddf4c5a305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:23.912421 31310 tablet_service.cc:1467] Processing CreateTablet for tablet c9d0511b167344ecae8a20ddf4c5a305 (DEFAULT_TABLE table=test-table [id=d929ea0d50a74ea492e05dab6775ee64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:23.912663 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d0511b167344ecae8a20ddf4c5a305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:23.914152 31566 tablet_bootstrap.cc:492] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6: Bootstrap starting.
I20260807 08:42:23.914335 31565 tablet_bootstrap.cc:492] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422: Bootstrap starting.
I20260807 08:42:23.914336 31612 ts_tablet_manager.cc:1918] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:23.914580 31612 log.cc:1196] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/ea3ec0e1c7c94315ae8999be84f06f65
I20260807 08:42:23.914842 31565 tablet_bootstrap.cc:654] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:23.914892 31566 tablet_bootstrap.cc:654] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:23.915160 31612 ts_tablet_manager.cc:1939] T ea3ec0e1c7c94315ae8999be84f06f65 P 4464d44169ee486d8cc524d0aac36422: Deleting consensus metadata
I20260807 08:42:23.915293 31567 tablet_bootstrap.cc:492] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5: Bootstrap starting.
I20260807 08:42:23.915704 31567 tablet_bootstrap.cc:654] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:23.915889 30983 catalog_manager.cc:4841] TS 4464d44169ee486d8cc524d0aac36422 (127.28.108.131:43827): tablet ea3ec0e1c7c94315ae8999be84f06f65 (table test-table [id=32d63318fa02417c85af887ac736f08d]) successfully deleted
I20260807 08:42:23.911248 31613 tablet_replica.cc:330] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: stopping tablet replica
I20260807 08:42:23.916132 31613 raft_consensus.cc:2229] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:23.916277 31613 raft_consensus.cc:2258] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:23.916760 31613 ts_tablet_manager.cc:1905] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:23.916947 31565 tablet_bootstrap.cc:492] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422: No bootstrap required, opened a new log
I20260807 08:42:23.917007 31565 ts_tablet_manager.cc:1397] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 08:42:23.917196 31565 raft_consensus.cc:348] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.917282 31565 raft_consensus.cc:374] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:23.917304 31565 raft_consensus.cc:729] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4464d44169ee486d8cc524d0aac36422, State: Initialized, Role: FOLLOWER
I20260807 08:42:23.917351 31565 consensus_queue.cc:260] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.917441 31565 ts_tablet_manager.cc:1428] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:23.917820 31567 tablet_bootstrap.cc:492] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5: No bootstrap required, opened a new log
I20260807 08:42:23.917882 31567 ts_tablet_manager.cc:1397] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260807 08:42:23.917972 31567 raft_consensus.cc:348] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.918026 31567 raft_consensus.cc:374] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:23.918043 31567 raft_consensus.cc:729] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22fd5c074e24a4f98ae251be8d8c7f5, State: Initialized, Role: FOLLOWER
I20260807 08:42:23.918093 31567 consensus_queue.cc:260] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.918167 31567 ts_tablet_manager.cc:1428] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:23.918967 31566 tablet_bootstrap.cc:492] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6: No bootstrap required, opened a new log
I20260807 08:42:23.919034 31566 ts_tablet_manager.cc:1397] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260807 08:42:23.919231 31566 raft_consensus.cc:348] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.919324 31566 raft_consensus.cc:374] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:23.919350 31566 raft_consensus.cc:729] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b437a4571b4dd1b24fca117406c1f6, State: Initialized, Role: FOLLOWER
I20260807 08:42:23.919364 31613 ts_tablet_manager.cc:1918] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:23.919397 31566 consensus_queue.cc:260] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:23.919457 31613 log.cc:1196] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/ea3ec0e1c7c94315ae8999be84f06f65
I20260807 08:42:23.919480 31566 ts_tablet_manager.cc:1428] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 08:42:23.919749 31613 ts_tablet_manager.cc:1939] T ea3ec0e1c7c94315ae8999be84f06f65 P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting consensus metadata
I20260807 08:42:23.920228 30981 catalog_manager.cc:4841] TS d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130:41993): tablet ea3ec0e1c7c94315ae8999be84f06f65 (table test-table [id=32d63318fa02417c85af887ac736f08d]) successfully deleted
I20260807 08:42:23.921783 31614 tablet_replica.cc:330] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: stopping tablet replica
I20260807 08:42:23.921943 31614 raft_consensus.cc:2229] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:23.922024 31614 raft_consensus.cc:2258] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:23.922616 31614 ts_tablet_manager.cc:1905] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:23.924854 31614 ts_tablet_manager.cc:1918] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:23.924960 31614 log.cc:1196] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/wal/wals/ea3ec0e1c7c94315ae8999be84f06f65
I20260807 08:42:23.925295 31614 ts_tablet_manager.cc:1939] T ea3ec0e1c7c94315ae8999be84f06f65 P d9b437a4571b4dd1b24fca117406c1f6: Deleting consensus metadata
I20260807 08:42:23.925987 30982 catalog_manager.cc:4841] TS d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731): tablet ea3ec0e1c7c94315ae8999be84f06f65 (table test-table [id=32d63318fa02417c85af887ac736f08d]) successfully deleted
I20260807 08:42:24.013095 31572 raft_consensus.cc:482] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:24.013234 31572 raft_consensus.cc:504] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:24.013813 31572 leader_election.cc:288] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130:41993), 4464d44169ee486d8cc524d0aac36422 (127.28.108.131:43827)
I20260807 08:42:24.018075 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:24.014037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:43827 (local address 127.28.108.129:33889)
0807 08:42:24.014174 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:24.014187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:24.014510 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:24.014684 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:24.014685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:24.014687 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:24.014850 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:24.014854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.015948 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.015948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:24.016931 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:24.016934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.017716 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.017717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:24.017815 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:24.017821 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:42:24.017915 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 08:42:24.017962 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":41,"thread_start_us":69,"threads_started":1}
I20260807 08:42:24.018075 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:24.014145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:33889 (local address 127.28.108.131:43827)
0807 08:42:24.014195 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:24.014197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:24.014503 (+ 306us) server_negotiation.cc:407] Connection header received
0807 08:42:24.014573 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:24.014573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:24.014584 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:24.014642 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:24.014967 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.015833 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.017027 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.017599 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.017708 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:24.017719 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:24.017922 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 08:42:24.017983 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 08:42:24.018657 31489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d0511b167344ecae8a20ddf4c5a305" candidate_uuid: "d9b437a4571b4dd1b24fca117406c1f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4464d44169ee486d8cc524d0aac36422" is_pre_election: true
I20260807 08:42:24.018776 31489 raft_consensus.cc:2454] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9b437a4571b4dd1b24fca117406c1f6 in term 0.
I20260807 08:42:24.018767 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:24.014235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:33463 (local address 127.28.108.130:41993)
0807 08:42:24.014278 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:24.014280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:24.014654 (+ 374us) server_negotiation.cc:407] Connection header received
0807 08:42:24.014804 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:24.014805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:24.014818 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:24.014947 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:24.015125 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.015833 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.017358 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.017987 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.018060 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:24.018071 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:24.018642 (+ 571us) server_negotiation.cc:299] Negotiation successful
0807 08:42:24.018692 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 08:42:24.018801 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:24.014037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41993 (local address 127.28.108.129:33463)
0807 08:42:24.014367 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:24.014378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:24.014670 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:24.014955 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:24.014956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:24.014958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:24.015038 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:24.015040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.015942 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.015943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:24.017267 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:24.017270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.018109 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.018110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:24.018568 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:24.018574 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:42:24.018632 (+ 58us) client_negotiation.cc:240] Negotiation successful
0807 08:42:24.018677 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":48,"thread_start_us":55,"threads_started":1}
I20260807 08:42:24.019054 31094 leader_election.cc:302] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4464d44169ee486d8cc524d0aac36422, d9b437a4571b4dd1b24fca117406c1f6; no voters:
I20260807 08:42:24.019274 31572 raft_consensus.cc:2781] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:24.019343 31572 raft_consensus.cc:482] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:24.019368 31572 raft_consensus.cc:3037] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:24.019342 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d0511b167344ecae8a20ddf4c5a305" candidate_uuid: "d9b437a4571b4dd1b24fca117406c1f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" is_pre_election: true
I20260807 08:42:24.019502 31332 raft_consensus.cc:2454] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9b437a4571b4dd1b24fca117406c1f6 in term 0.
I20260807 08:42:24.020313 31572 raft_consensus.cc:504] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:24.020483 31572 leader_election.cc:288] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [CANDIDATE]: Term 1 election: Requested vote from peers d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130:41993), 4464d44169ee486d8cc524d0aac36422 (127.28.108.131:43827)
I20260807 08:42:24.020695 31489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d0511b167344ecae8a20ddf4c5a305" candidate_uuid: "d9b437a4571b4dd1b24fca117406c1f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4464d44169ee486d8cc524d0aac36422"
I20260807 08:42:24.020781 31489 raft_consensus.cc:3037] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:24.020763 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d0511b167344ecae8a20ddf4c5a305" candidate_uuid: "d9b437a4571b4dd1b24fca117406c1f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5"
I20260807 08:42:24.020861 31332 raft_consensus.cc:3037] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:24.021608 31332 raft_consensus.cc:2454] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9b437a4571b4dd1b24fca117406c1f6 in term 1.
I20260807 08:42:24.021853 31092 leader_election.cc:302] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d22fd5c074e24a4f98ae251be8d8c7f5, d9b437a4571b4dd1b24fca117406c1f6; no voters:
I20260807 08:42:24.021979 31572 raft_consensus.cc:2781] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:24.022059 31489 raft_consensus.cc:2454] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9b437a4571b4dd1b24fca117406c1f6 in term 1.
I20260807 08:42:24.022184 31572 raft_consensus.cc:686] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 LEADER]: Becoming Leader. State: Replica: d9b437a4571b4dd1b24fca117406c1f6, State: Running, Role: LEADER
I20260807 08:42:24.022270 31572 consensus_queue.cc:237] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:24.023149 30998 catalog_manager.cc:5495] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129). New cstate: current_term: 1 leader_uuid: "d9b437a4571b4dd1b24fca117406c1f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } health_report { overall_health: UNKNOWN } } }
I20260807 08:42:24.041757 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:24.038082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.28.108.129:44731)
0807 08:42:24.038135 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:24.038137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:24.038147 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:24.038355 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:24.038356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:24.038383 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:24.038474 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:24.038623 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.039406 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.040423 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.040738 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.040818 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:24.041079 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:24.041326 (+ 247us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:24.041480 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:24.041593 (+ 113us) server_negotiation.cc:299] Negotiation successful
0807 08:42:24.041657 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 08:42:24.048935 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:24.045602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:41119 (local address 127.28.108.130:41993)
0807 08:42:24.045681 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:24.045684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:24.045803 (+ 119us) server_negotiation.cc:407] Connection header received
0807 08:42:24.045857 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:24.045858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:24.045870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:24.045938 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:24.046082 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.046756 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.047902 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.048238 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.048311 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:24.048612 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:24.048654 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:24.048705 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:24.048820 (+ 115us) server_negotiation.cc:299] Negotiation successful
0807 08:42:24.048846 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 08:42:24.049278 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:24.045485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41993 (local address 127.28.108.129:41119)
0807 08:42:24.045718 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:24.045731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:24.045805 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:24.045940 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:24.045941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:24.045942 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:24.046022 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:24.046023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.046862 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.046863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:24.047808 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:24.047812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.048350 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.048351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:24.048417 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:24.048437 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:24.048747 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:24.048750 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:24.048799 (+ 49us) client_negotiation.cc:240] Negotiation successful
0807 08:42:24.048858 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":113,"threads_started":1}
I20260807 08:42:24.052117 31332 raft_consensus.cc:1261] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Refusing update from remote peer d9b437a4571b4dd1b24fca117406c1f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:24.052120 31489 raft_consensus.cc:1261] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Refusing update from remote peer d9b437a4571b4dd1b24fca117406c1f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:24.052605 31620 consensus_queue.cc:1035] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:24.052778 31572 consensus_queue.cc:1035] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:24.061426 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:24.058751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:44731 (local address 127.28.108.130:43473)
0807 08:42:24.058819 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:24.058831 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:24.058883 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:24.059166 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:24.059167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:24.059168 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:24.059217 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:24.059219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.059971 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.059972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:24.060637 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:24.060640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.060994 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.060995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:24.061052 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:24.061059 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:24.061250 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:24.061253 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:24.061307 (+ 54us) client_negotiation.cc:240] Negotiation successful
0807 08:42:24.061357 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260807 08:42:24.061417 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:24.058945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:43473 (local address 127.28.108.129:44731)
0807 08:42:24.059003 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:24.059005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:24.059013 (+ 8us) server_negotiation.cc:407] Connection header received
0807 08:42:24.059037 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:24.059038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:24.059049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:24.059112 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:24.059311 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.059891 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.060706 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:24.060919 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:24.060997 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:24.061143 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:24.061179 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:24.061249 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:24.061312 (+ 63us) server_negotiation.cc:299] Negotiation successful
0807 08:42:24.061334 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260807 08:42:24.061812 31593 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 08:42:25.591017 30998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:25.591171 30998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:25.592257 30998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb: Sending DeleteTablet for 3 replicas of tablet c9d0511b167344ecae8a20ddf4c5a305
I20260807 08:42:25.592651 31469 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d0511b167344ecae8a20ddf4c5a305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:25 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:58656
I20260807 08:42:25.592664 31310 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d0511b167344ecae8a20ddf4c5a305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:25 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:54918
I20260807 08:42:25.592681 31147 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d0511b167344ecae8a20ddf4c5a305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:25 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:38568
I20260807 08:42:25.593084 31657 tablet_replica.cc:330] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5: stopping tablet replica
I20260807 08:42:25.593235 31655 tablet_replica.cc:330] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422: stopping tablet replica
I20260807 08:42:25.593308 31655 raft_consensus.cc:2229] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:25.593369 31655 raft_consensus.cc:2258] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:25.593470 31657 raft_consensus.cc:2229] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:25.593567 31657 raft_consensus.cc:2258] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:25.593840 31655 ts_tablet_manager.cc:1905] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:25.596122 31657 ts_tablet_manager.cc:1905] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:25.596688 31655 ts_tablet_manager.cc:1918] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 08:42:25.596891 31655 log.cc:1196] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/c9d0511b167344ecae8a20ddf4c5a305
I20260807 08:42:25.597241 31655 ts_tablet_manager.cc:1939] T c9d0511b167344ecae8a20ddf4c5a305 P 4464d44169ee486d8cc524d0aac36422: Deleting consensus metadata
I20260807 08:42:25.597754 30983 catalog_manager.cc:4841] TS 4464d44169ee486d8cc524d0aac36422 (127.28.108.131:43827): tablet c9d0511b167344ecae8a20ddf4c5a305 (table test-table [id=d929ea0d50a74ea492e05dab6775ee64]) successfully deleted
I20260807 08:42:25.598483 31657 ts_tablet_manager.cc:1918] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 08:42:25.598596 31657 log.cc:1196] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/c9d0511b167344ecae8a20ddf4c5a305
I20260807 08:42:25.598882 31657 ts_tablet_manager.cc:1939] T c9d0511b167344ecae8a20ddf4c5a305 P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting consensus metadata
I20260807 08:42:25.599316 30981 catalog_manager.cc:4841] TS d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130:41993): tablet c9d0511b167344ecae8a20ddf4c5a305 (table test-table [id=d929ea0d50a74ea492e05dab6775ee64]) successfully deleted
I20260807 08:42:25.601186 31656 tablet_replica.cc:330] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6: stopping tablet replica
I20260807 08:42:25.601296 31656 raft_consensus.cc:2229] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:25.601442 31656 raft_consensus.cc:2258] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:25.601788 31656 ts_tablet_manager.cc:1905] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:25.603595 31656 ts_tablet_manager.cc:1918] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 08:42:25.603768 31656 log.cc:1196] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/wal/wals/c9d0511b167344ecae8a20ddf4c5a305
I20260807 08:42:25.604055 31656 ts_tablet_manager.cc:1939] T c9d0511b167344ecae8a20ddf4c5a305 P d9b437a4571b4dd1b24fca117406c1f6: Deleting consensus metadata
I20260807 08:42:25.604516 30982 catalog_manager.cc:4841] TS d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731): tablet c9d0511b167344ecae8a20ddf4c5a305 (table test-table [id=d929ea0d50a74ea492e05dab6775ee64]) successfully deleted
Aug 07 08:42:25 dist-test-slave-9zdj krb5kdc[30948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092145, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 08:42:25 dist-test-slave-9zdj krb5kdc[30948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092145, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:25.630064 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:25.615084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.28.108.190:36229)
0807 08:42:25.615360 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:25.615362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:25.615372 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:25.615399 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:25.615400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:25.615411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:25.615478 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:25.616094 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.616666 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.617897 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.618326 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.618452 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:25.626779 (+ 8327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:25.626781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:25.626782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:25.626791 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:25.628566 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:25.628996 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:25.628997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:25.628998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:25.629064 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:25.629392 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:25.629394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:25.629394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:25.629529 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:25.629675 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:25.629896 (+ 221us) server_negotiation.cc:299] Negotiation successful
0807 08:42:25.629984 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":213,"threads_started":1}
I20260807 08:42:25.632483 30998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:25.632658 30998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:25.634883 31310 tablet_service.cc:1467] Processing CreateTablet for tablet eeaf9d2b33814bfb9ab0f0cbdacc695f (DEFAULT_TABLE table=test-table [id=f7e59d16ded74ed4b1f7cd804219235e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:25.634908 31147 tablet_service.cc:1467] Processing CreateTablet for tablet eeaf9d2b33814bfb9ab0f0cbdacc695f (DEFAULT_TABLE table=test-table [id=f7e59d16ded74ed4b1f7cd804219235e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:25.635069 31147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaf9d2b33814bfb9ab0f0cbdacc695f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:25.635135 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaf9d2b33814bfb9ab0f0cbdacc695f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:25.636385 31666 tablet_bootstrap.cc:492] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6: Bootstrap starting.
I20260807 08:42:25.636477 31667 tablet_bootstrap.cc:492] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5: Bootstrap starting.
I20260807 08:42:25.636672 31469 tablet_service.cc:1467] Processing CreateTablet for tablet eeaf9d2b33814bfb9ab0f0cbdacc695f (DEFAULT_TABLE table=test-table [id=f7e59d16ded74ed4b1f7cd804219235e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:25.636824 31469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaf9d2b33814bfb9ab0f0cbdacc695f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:25.636893 31666 tablet_bootstrap.cc:654] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:25.636946 31667 tablet_bootstrap.cc:654] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:25.637566 31666 tablet_bootstrap.cc:492] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6: No bootstrap required, opened a new log
I20260807 08:42:25.637605 31666 ts_tablet_manager.cc:1397] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:25.637799 31666 raft_consensus.cc:348] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.637903 31666 raft_consensus.cc:374] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:25.637954 31666 raft_consensus.cc:729] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b437a4571b4dd1b24fca117406c1f6, State: Initialized, Role: FOLLOWER
I20260807 08:42:25.638036 31666 consensus_queue.cc:260] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.638070 31667 tablet_bootstrap.cc:492] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5: No bootstrap required, opened a new log
I20260807 08:42:25.638134 31667 ts_tablet_manager.cc:1397] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 08:42:25.638250 31667 raft_consensus.cc:348] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.638338 31667 raft_consensus.cc:374] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:25.638360 31667 raft_consensus.cc:729] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22fd5c074e24a4f98ae251be8d8c7f5, State: Initialized, Role: FOLLOWER
I20260807 08:42:25.638413 31667 consensus_queue.cc:260] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.638520 31667 ts_tablet_manager.cc:1428] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:25.638680 31666 ts_tablet_manager.cc:1428] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 08:42:25.638835 31670 tablet_bootstrap.cc:492] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422: Bootstrap starting.
I20260807 08:42:25.639303 31670 tablet_bootstrap.cc:654] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:25.639963 31670 tablet_bootstrap.cc:492] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422: No bootstrap required, opened a new log
I20260807 08:42:25.640009 31670 ts_tablet_manager.cc:1397] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:25.640105 31670 raft_consensus.cc:348] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.640161 31670 raft_consensus.cc:374] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:25.640174 31670 raft_consensus.cc:729] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4464d44169ee486d8cc524d0aac36422, State: Initialized, Role: FOLLOWER
I20260807 08:42:25.640242 31670 consensus_queue.cc:260] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.640496 31670 ts_tablet_manager.cc:1428] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 08:42:25.789335 31673 raft_consensus.cc:482] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:25.789485 31673 raft_consensus.cc:504] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.789985 31673 leader_election.cc:288] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731), d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130:41993)
I20260807 08:42:25.794734 31674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:25.790215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41993 (local address 127.28.108.131:44515)
0807 08:42:25.790445 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:25.790460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:25.790527 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:25.790924 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:25.790926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:25.790929 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:25.791096 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:25.791099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.792155 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.792155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:25.793653 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:25.793658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.794354 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.794356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:25.794480 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:25.794486 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:42:25.794553 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 08:42:25.794614 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":84,"thread_start_us":141,"threads_started":1}
I20260807 08:42:25.794806 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:25.790336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:44515 (local address 127.28.108.130:41993)
0807 08:42:25.790518 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:25.790521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:25.790536 (+ 15us) server_negotiation.cc:407] Connection header received
0807 08:42:25.790609 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:25.790610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:25.790623 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:25.790723 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:25.791208 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.792049 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.793798 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.794236 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.794355 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:25.794366 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:25.794626 (+ 260us) server_negotiation.cc:299] Negotiation successful
0807 08:42:25.794705 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":108,"threads_started":1}
I20260807 08:42:25.794806 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:25.790406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:37865 (local address 127.28.108.129:44731)
0807 08:42:25.790583 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:25.790585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:25.790630 (+ 45us) server_negotiation.cc:407] Connection header received
0807 08:42:25.790711 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:25.790712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:25.790724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:25.790795 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:25.791208 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.792002 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.793796 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.794375 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.794452 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:25.794460 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:25.794669 (+ 209us) server_negotiation.cc:299] Negotiation successful
0807 08:42:25.794730 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":110,"threads_started":1}
I20260807 08:42:25.794746 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:25.790214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:44731 (local address 127.28.108.131:37865)
0807 08:42:25.790576 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:25.790586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:25.790644 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:25.790936 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:25.790937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:25.790938 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:25.791096 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:25.791099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.792153 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.792155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:25.793682 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:25.793684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.794485 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.794486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:25.794557 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:25.794560 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:25.794614 (+ 54us) client_negotiation.cc:240] Negotiation successful
0807 08:42:25.794647 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":121,"spinlock_wait_cycles":896,"thread_start_us":143,"threads_started":1}
I20260807 08:42:25.795362 31167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf9d2b33814bfb9ab0f0cbdacc695f" candidate_uuid: "4464d44169ee486d8cc524d0aac36422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b437a4571b4dd1b24fca117406c1f6" is_pre_election: true
I20260807 08:42:25.795523 31167 raft_consensus.cc:2454] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4464d44169ee486d8cc524d0aac36422 in term 0.
I20260807 08:42:25.795593 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf9d2b33814bfb9ab0f0cbdacc695f" candidate_uuid: "4464d44169ee486d8cc524d0aac36422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" is_pre_election: true
I20260807 08:42:25.795717 31332 raft_consensus.cc:2454] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4464d44169ee486d8cc524d0aac36422 in term 0.
I20260807 08:42:25.795776 31419 leader_election.cc:302] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4464d44169ee486d8cc524d0aac36422, d9b437a4571b4dd1b24fca117406c1f6; no voters:
I20260807 08:42:25.796006 31673 raft_consensus.cc:2781] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:25.796063 31673 raft_consensus.cc:482] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:25.796082 31673 raft_consensus.cc:3037] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:25.797154 31673 raft_consensus.cc:504] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.797336 31673 leader_election.cc:288] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [CANDIDATE]: Term 1 election: Requested vote from peers d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731), d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130:41993)
I20260807 08:42:25.797557 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf9d2b33814bfb9ab0f0cbdacc695f" candidate_uuid: "4464d44169ee486d8cc524d0aac36422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5"
I20260807 08:42:25.797557 31167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf9d2b33814bfb9ab0f0cbdacc695f" candidate_uuid: "4464d44169ee486d8cc524d0aac36422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b437a4571b4dd1b24fca117406c1f6"
I20260807 08:42:25.797679 31332 raft_consensus.cc:3037] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:25.797679 31167 raft_consensus.cc:3037] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:25.798588 31332 raft_consensus.cc:2454] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4464d44169ee486d8cc524d0aac36422 in term 1.
I20260807 08:42:25.798671 31167 raft_consensus.cc:2454] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4464d44169ee486d8cc524d0aac36422 in term 1.
I20260807 08:42:25.798913 31419 leader_election.cc:302] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4464d44169ee486d8cc524d0aac36422, d9b437a4571b4dd1b24fca117406c1f6; no voters:
I20260807 08:42:25.799021 31673 raft_consensus.cc:2781] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:25.799208 31673 raft_consensus.cc:686] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 1 LEADER]: Becoming Leader. State: Replica: 4464d44169ee486d8cc524d0aac36422, State: Running, Role: LEADER
I20260807 08:42:25.799429 31673 consensus_queue.cc:237] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.800251 30998 catalog_manager.cc:5495] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4464d44169ee486d8cc524d0aac36422 (127.28.108.131). New cstate: current_term: 1 leader_uuid: "4464d44169ee486d8cc524d0aac36422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } health_report { overall_health: HEALTHY } } }
I20260807 08:42:25.830538 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:25.827483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.28.108.131:43827)
0807 08:42:25.827673 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:25.827674 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:25.827681 (+ 7us) server_negotiation.cc:407] Connection header received
0807 08:42:25.827727 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:25.827728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:25.827751 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:25.827855 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:25.828033 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.828607 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.829503 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.829770 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.829855 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:25.830109 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:25.830238 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:25.830339 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:25.830382 (+ 43us) server_negotiation.cc:299] Negotiation successful
0807 08:42:25.830407 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260807 08:42:25.836902 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:25.834256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:41089 (local address 127.28.108.130:41993)
0807 08:42:25.834344 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:25.834346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:25.834363 (+ 17us) server_negotiation.cc:407] Connection header received
0807 08:42:25.834390 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:25.834390 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:25.834403 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:25.834456 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:25.834622 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.835156 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.836006 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.836211 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.836269 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:25.836602 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:25.836638 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:25.836686 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:25.836795 (+ 109us) server_negotiation.cc:299] Negotiation successful
0807 08:42:25.836819 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 08:42:25.836917 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:25.834016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41993 (local address 127.28.108.131:41089)
0807 08:42:25.834227 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:25.834241 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:25.834307 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:25.834484 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:25.834485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:25.834487 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:25.834555 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:25.834556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.835246 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.835247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:25.835917 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:25.835920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.836375 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.836377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:25.836456 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:25.836478 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:25.836717 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:25.836720 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:25.836775 (+ 55us) client_negotiation.cc:240] Negotiation successful
0807 08:42:25.836827 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
I20260807 08:42:25.840193 31332 raft_consensus.cc:1261] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Refusing update from remote peer 4464d44169ee486d8cc524d0aac36422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:25.840241 31167 raft_consensus.cc:1261] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Refusing update from remote peer 4464d44169ee486d8cc524d0aac36422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:25.840543 31673 consensus_queue.cc:1035] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:25.840741 31678 consensus_queue.cc:1035] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:25.849947 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:25.846950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:45553 (local address 127.28.108.131:43827)
0807 08:42:25.846999 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:25.847001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:25.847180 (+ 179us) server_negotiation.cc:407] Connection header received
0807 08:42:25.847262 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:25.847263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:25.847277 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:25.847317 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:25.847547 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.848125 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.848995 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.849216 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.849301 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:25.849486 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:25.849549 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:25.849641 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:25.849824 (+ 183us) server_negotiation.cc:299] Negotiation successful
0807 08:42:25.849859 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:42:25.849956 31690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:25.846914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:43827 (local address 127.28.108.130:45553)
0807 08:42:25.847072 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:25.847082 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:25.847174 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:25.847408 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:25.847409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:25.847411 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:25.847473 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:25.847475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.848222 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.848224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:25.848921 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:25.848925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:25.849301 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:25.849302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:25.849374 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:25.849381 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:25.849733 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:25.849736 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:25.849796 (+ 60us) client_negotiation.cc:240] Negotiation successful
0807 08:42:25.849861 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":107,"threads_started":1}
I20260807 08:42:25.861270 30996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:25.861438 30996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:25.862308 30996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb: Sending DeleteTablet for 3 replicas of tablet eeaf9d2b33814bfb9ab0f0cbdacc695f
I20260807 08:42:25.862686 31147 tablet_service.cc:1514] Processing DeleteTablet for tablet eeaf9d2b33814bfb9ab0f0cbdacc695f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:25 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:38568
I20260807 08:42:25.862695 31310 tablet_service.cc:1514] Processing DeleteTablet for tablet eeaf9d2b33814bfb9ab0f0cbdacc695f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:25 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:54918
I20260807 08:42:25.862838 31656 tablet_replica.cc:330] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6: stopping tablet replica
I20260807 08:42:25.862888 31469 tablet_service.cc:1514] Processing DeleteTablet for tablet eeaf9d2b33814bfb9ab0f0cbdacc695f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:25 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:58656
I20260807 08:42:25.862915 31656 raft_consensus.cc:2229] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:25.862967 31656 raft_consensus.cc:2258] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:25.862982 31655 tablet_replica.cc:330] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422: stopping tablet replica
I20260807 08:42:25.863057 31655 raft_consensus.cc:2229] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:25.863054 31657 tablet_replica.cc:330] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5: stopping tablet replica
I20260807 08:42:25.863137 31657 raft_consensus.cc:2229] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:25.863157 31655 raft_consensus.cc:2258] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:25.863197 31657 raft_consensus.cc:2258] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:25.863413 31656 ts_tablet_manager.cc:1905] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:25.863476 31655 ts_tablet_manager.cc:1905] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:25.863832 30996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:25.864004 30996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:25.865362 31147 tablet_service.cc:1467] Processing CreateTablet for tablet b7fb8d0f4c274e3f88b33eabb0607b79 (DEFAULT_TABLE table=test-table [id=dfc98415395b44688e377cc9fc929724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:25.865417 31469 tablet_service.cc:1467] Processing CreateTablet for tablet b7fb8d0f4c274e3f88b33eabb0607b79 (DEFAULT_TABLE table=test-table [id=dfc98415395b44688e377cc9fc929724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:25.865553 31147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fb8d0f4c274e3f88b33eabb0607b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:25.865545 31310 tablet_service.cc:1467] Processing CreateTablet for tablet b7fb8d0f4c274e3f88b33eabb0607b79 (DEFAULT_TABLE table=test-table [id=dfc98415395b44688e377cc9fc929724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:25.865579 31656 ts_tablet_manager.cc:1918] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:25.865664 31656 log.cc:1196] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-0/wal/wals/eeaf9d2b33814bfb9ab0f0cbdacc695f
I20260807 08:42:25.865686 31310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fb8d0f4c274e3f88b33eabb0607b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:25.865890 31657 ts_tablet_manager.cc:1905] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:25.865918 31656 ts_tablet_manager.cc:1939] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d9b437a4571b4dd1b24fca117406c1f6: Deleting consensus metadata
I20260807 08:42:25.866331 30982 catalog_manager.cc:4841] TS d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731): tablet eeaf9d2b33814bfb9ab0f0cbdacc695f (table test-table [id=f7e59d16ded74ed4b1f7cd804219235e]) successfully deleted
I20260807 08:42:25.867247 31667 tablet_bootstrap.cc:492] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5: Bootstrap starting.
I20260807 08:42:25.867547 31666 tablet_bootstrap.cc:492] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6: Bootstrap starting.
I20260807 08:42:25.867573 31655 ts_tablet_manager.cc:1918] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:25.867619 31657 ts_tablet_manager.cc:1918] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:25.867636 31655 log.cc:1196] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/eeaf9d2b33814bfb9ab0f0cbdacc695f
I20260807 08:42:25.867686 31657 log.cc:1196] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/eeaf9d2b33814bfb9ab0f0cbdacc695f
I20260807 08:42:25.867905 31655 ts_tablet_manager.cc:1939] T eeaf9d2b33814bfb9ab0f0cbdacc695f P 4464d44169ee486d8cc524d0aac36422: Deleting consensus metadata
I20260807 08:42:25.867765 31667 tablet_bootstrap.cc:654] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:25.868027 31666 tablet_bootstrap.cc:654] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:25.868077 31469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fb8d0f4c274e3f88b33eabb0607b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:25.868076 31657 ts_tablet_manager.cc:1939] T eeaf9d2b33814bfb9ab0f0cbdacc695f P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting consensus metadata
I20260807 08:42:25.868624 31666 tablet_bootstrap.cc:492] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6: No bootstrap required, opened a new log
I20260807 08:42:25.868682 31666 ts_tablet_manager.cc:1397] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:25.868808 31666 raft_consensus.cc:348] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.868884 31666 raft_consensus.cc:374] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:25.868908 31666 raft_consensus.cc:729] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b437a4571b4dd1b24fca117406c1f6, State: Initialized, Role: FOLLOWER
I20260807 08:42:25.868969 31666 consensus_queue.cc:260] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.869087 31666 ts_tablet_manager.cc:1428] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:25.869197 30981 catalog_manager.cc:4841] TS d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130:41993): tablet eeaf9d2b33814bfb9ab0f0cbdacc695f (table test-table [id=f7e59d16ded74ed4b1f7cd804219235e]) successfully deleted
I20260807 08:42:25.869603 31670 tablet_bootstrap.cc:492] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422: Bootstrap starting.
I20260807 08:42:25.870085 31670 tablet_bootstrap.cc:654] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:25.870165 30983 catalog_manager.cc:4841] TS 4464d44169ee486d8cc524d0aac36422 (127.28.108.131:43827): tablet eeaf9d2b33814bfb9ab0f0cbdacc695f (table test-table [id=f7e59d16ded74ed4b1f7cd804219235e]) successfully deleted
I20260807 08:42:25.870493 31667 tablet_bootstrap.cc:492] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5: No bootstrap required, opened a new log
I20260807 08:42:25.870554 31667 ts_tablet_manager.cc:1397] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 08:42:25.870759 31670 tablet_bootstrap.cc:492] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422: No bootstrap required, opened a new log
I20260807 08:42:25.870711 31667 raft_consensus.cc:348] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.870795 31667 raft_consensus.cc:374] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:25.870803 31670 ts_tablet_manager.cc:1397] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:25.870820 31667 raft_consensus.cc:729] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22fd5c074e24a4f98ae251be8d8c7f5, State: Initialized, Role: FOLLOWER
I20260807 08:42:25.870870 31667 consensus_queue.cc:260] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.870970 31667 ts_tablet_manager.cc:1428] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:25.870975 31670 raft_consensus.cc:348] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.871039 31670 raft_consensus.cc:374] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:25.871066 31670 raft_consensus.cc:729] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4464d44169ee486d8cc524d0aac36422, State: Initialized, Role: FOLLOWER
I20260807 08:42:25.871111 31670 consensus_queue.cc:260] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.871227 31670 ts_tablet_manager.cc:1428] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 08:42:25.890709 31673 raft_consensus.cc:482] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:25.890851 31673 raft_consensus.cc:504] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.891045 31673 leader_election.cc:288] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731), d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130:41993)
I20260807 08:42:25.891252 31167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fb8d0f4c274e3f88b33eabb0607b79" candidate_uuid: "4464d44169ee486d8cc524d0aac36422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b437a4571b4dd1b24fca117406c1f6" is_pre_election: true
I20260807 08:42:25.891287 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fb8d0f4c274e3f88b33eabb0607b79" candidate_uuid: "4464d44169ee486d8cc524d0aac36422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" is_pre_election: true
I20260807 08:42:25.891373 31167 raft_consensus.cc:2454] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4464d44169ee486d8cc524d0aac36422 in term 0.
I20260807 08:42:25.891395 31332 raft_consensus.cc:2454] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4464d44169ee486d8cc524d0aac36422 in term 0.
I20260807 08:42:25.891528 31419 leader_election.cc:302] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4464d44169ee486d8cc524d0aac36422, d9b437a4571b4dd1b24fca117406c1f6; no voters:
I20260807 08:42:25.891640 31673 raft_consensus.cc:2781] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:25.891717 31673 raft_consensus.cc:482] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:25.891750 31673 raft_consensus.cc:3037] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:25.892534 31673 raft_consensus.cc:504] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.892700 31673 leader_election.cc:288] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [CANDIDATE]: Term 1 election: Requested vote from peers d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731), d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130:41993)
I20260807 08:42:25.892925 31167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fb8d0f4c274e3f88b33eabb0607b79" candidate_uuid: "4464d44169ee486d8cc524d0aac36422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b437a4571b4dd1b24fca117406c1f6"
I20260807 08:42:25.892940 31332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fb8d0f4c274e3f88b33eabb0607b79" candidate_uuid: "4464d44169ee486d8cc524d0aac36422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5"
I20260807 08:42:25.893014 31167 raft_consensus.cc:3037] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:25.893034 31332 raft_consensus.cc:3037] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:25.893766 31332 raft_consensus.cc:2454] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4464d44169ee486d8cc524d0aac36422 in term 1.
I20260807 08:42:25.893883 31167 raft_consensus.cc:2454] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4464d44169ee486d8cc524d0aac36422 in term 1.
I20260807 08:42:25.893989 31417 leader_election.cc:302] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4464d44169ee486d8cc524d0aac36422, d22fd5c074e24a4f98ae251be8d8c7f5; no voters:
I20260807 08:42:25.894079 31673 raft_consensus.cc:2781] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:25.894136 31673 raft_consensus.cc:686] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 1 LEADER]: Becoming Leader. State: Replica: 4464d44169ee486d8cc524d0aac36422, State: Running, Role: LEADER
I20260807 08:42:25.894192 31673 consensus_queue.cc:237] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } }
I20260807 08:42:25.894858 30996 catalog_manager.cc:5495] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4464d44169ee486d8cc524d0aac36422 (127.28.108.131). New cstate: current_term: 1 leader_uuid: "4464d44169ee486d8cc524d0aac36422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4464d44169ee486d8cc524d0aac36422" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43827 } health_report { overall_health: HEALTHY } } }
I20260807 08:42:25.903580 31168 raft_consensus.cc:1261] T b7fb8d0f4c274e3f88b33eabb0607b79 P d9b437a4571b4dd1b24fca117406c1f6 [term 1 FOLLOWER]: Refusing update from remote peer 4464d44169ee486d8cc524d0aac36422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:25.903827 31332 raft_consensus.cc:1261] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Refusing update from remote peer 4464d44169ee486d8cc524d0aac36422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:25.903831 31673 consensus_queue.cc:1035] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b437a4571b4dd1b24fca117406c1f6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:25.904079 31673 consensus_queue.cc:1035] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d22fd5c074e24a4f98ae251be8d8c7f5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 08:42:25.908747 31634 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 08:42:27.429577 30998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:27.429772 30998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:27.430824 30998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d7d7f7d04af4c81854181e0863f4fdb: Sending DeleteTablet for 3 replicas of tablet b7fb8d0f4c274e3f88b33eabb0607b79
I20260807 08:42:27.431151 31469 tablet_service.cc:1514] Processing DeleteTablet for tablet b7fb8d0f4c274e3f88b33eabb0607b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:27 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:58656
I20260807 08:42:27.431165 31147 tablet_service.cc:1514] Processing DeleteTablet for tablet b7fb8d0f4c274e3f88b33eabb0607b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:27 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:38568
I20260807 08:42:27.431165 31310 tablet_service.cc:1514] Processing DeleteTablet for tablet b7fb8d0f4c274e3f88b33eabb0607b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:27 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:54918
I20260807 08:42:27.431421 31720 tablet_replica.cc:330] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422: stopping tablet replica
I20260807 08:42:27.431324 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 31076
I20260807 08:42:27.434741 31722 tablet_replica.cc:330] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5: stopping tablet replica
I20260807 08:42:27.435074 31722 raft_consensus.cc:2229] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:27.435207 31722 raft_consensus.cc:2258] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:27.435777 31722 ts_tablet_manager.cc:1905] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:27.431506 31720 raft_consensus.cc:2229] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:27.438867 31720 raft_consensus.cc:2258] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:27.438964 31722 ts_tablet_manager.cc:1918] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 08:42:27.439065 31722 log.cc:1196] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/b7fb8d0f4c274e3f88b33eabb0607b79
I20260807 08:42:27.439323 31720 ts_tablet_manager.cc:1905] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:27.439404 31722 ts_tablet_manager.cc:1939] T b7fb8d0f4c274e3f88b33eabb0607b79 P d22fd5c074e24a4f98ae251be8d8c7f5: Deleting consensus metadata
I20260807 08:42:27.439900 30981 catalog_manager.cc:4841] TS d22fd5c074e24a4f98ae251be8d8c7f5 (127.28.108.130:41993): tablet b7fb8d0f4c274e3f88b33eabb0607b79 (table test-table [id=dfc98415395b44688e377cc9fc929724]) successfully deleted
I20260807 08:42:27.441056 31720 ts_tablet_manager.cc:1918] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 08:42:27.441154 31720 log.cc:1196] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/b7fb8d0f4c274e3f88b33eabb0607b79
I20260807 08:42:27.441488 31720 ts_tablet_manager.cc:1939] T b7fb8d0f4c274e3f88b33eabb0607b79 P 4464d44169ee486d8cc524d0aac36422: Deleting consensus metadata
I20260807 08:42:27.442559 30983 catalog_manager.cc:4841] TS 4464d44169ee486d8cc524d0aac36422 (127.28.108.131:43827): tablet b7fb8d0f4c274e3f88b33eabb0607b79 (table test-table [id=dfc98415395b44688e377cc9fc929724]) successfully deleted
W20260807 08:42:27.445600 30982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:44731 (error 108)
I20260807 08:42:27.445789 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 31232
I20260807 08:42:27.448594 31724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:27.448331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:44731 (local address 127.0.0.1:48936)
0807 08:42:27.448482 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:27.448551 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:44731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":103,"threads_started":1}
W20260807 08:42:27.448766 30982 catalog_manager.cc:4584] TS d9b437a4571b4dd1b24fca117406c1f6 (127.28.108.129:44731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7fb8d0f4c274e3f88b33eabb0607b79: Network error: Client connection negotiation failed: client connection to 127.28.108.129:44731: connect: Connection refused (error 111)
I20260807 08:42:27.456874 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 31398
I20260807 08:42:27.467319 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 30964
2026-08-07T08:42:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6001 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:27 dist-test-slave-9zdj krb5kdc[31727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:27 dist-test-slave-9zdj krb5kdc[31727](info): set up 2 sockets
Aug 07 08:42:27 dist-test-slave-9zdj krb5kdc[31727](info): commencing operation
krb5kdc: starting...
W20260807 08:42:28.655906 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:28 dist-test-slave-9zdj krb5kdc[31727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:28.809237 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:33865
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:33865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:28.899147 31740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:28.899448 31740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:28.899494 31740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:28.900785 31740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:28.900887 31740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:28.900913 31740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:28.900934 31740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:28.900971 31740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:28.902771 31740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:33865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:33865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:28.903131 31740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:28.903524 31740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:28.909511 31745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:28.909511 31746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:28.909511 31748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:28.909577 31740 server_base.cc:1034] running on GCE node
I20260807 08:42:28.910251 31740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:28.910637 31740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:28.911898 31740 hybrid_clock.cc:648] HybridClock initialized: now 1786092148911888 us; error 59 us; skew 500 ppm
Aug 07 08:42:28 dist-test-slave-9zdj krb5kdc[31727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:28.915118 31740 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:28.915728 31740 webserver.cc:460] Webserver started at http://127.28.108.190:46289/ using document root <none> and password file <none>
I20260807 08:42:28.916013 31740 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:28.916090 31740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:28.916265 31740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:28.917523 31740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "ca7a8221339b4726842a82818d615874"
format_stamp: "Formatted at 2026-08-07 08:42:28 on dist-test-slave-9zdj"
server_key: "628135edc2271d5d7a790b4242819dbb"
server_key_iv: "e3a1874b039a24cabe0d0bf48bb570b5"
server_key_version: "encryptionkey@0"
I20260807 08:42:28.917994 31740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "ca7a8221339b4726842a82818d615874"
format_stamp: "Formatted at 2026-08-07 08:42:28 on dist-test-slave-9zdj"
server_key: "628135edc2271d5d7a790b4242819dbb"
server_key_iv: "e3a1874b039a24cabe0d0bf48bb570b5"
server_key_version: "encryptionkey@0"
I20260807 08:42:28.919586 31740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:42:28.920498 31757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:28.920703 31740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:28.920856 31740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "ca7a8221339b4726842a82818d615874"
format_stamp: "Formatted at 2026-08-07 08:42:28 on dist-test-slave-9zdj"
server_key: "628135edc2271d5d7a790b4242819dbb"
server_key_iv: "e3a1874b039a24cabe0d0bf48bb570b5"
server_key_version: "encryptionkey@0"
I20260807 08:42:28.920945 31740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:28.937222 31740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:28.939888 31740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:28.940085 31740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:28.944908 31740 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:33865
I20260807 08:42:28.945380 31740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:28.945540 31813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:33865 every 8 connection(s)
I20260807 08:42:28.946455 31814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:28.947001 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 31740
I20260807 08:42:28.947108 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:28.947302 29106 external_mini_cluster.cc:1428] Setting key 48ab1fc7e80d37775053216868abb791
I20260807 08:42:28.951874 31814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874: Bootstrap starting.
I20260807 08:42:28.952886 31814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:28.953382 31814 log.cc:824] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:28.954519 31814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874: No bootstrap required, opened a new log
Aug 07 08:42:28 dist-test-slave-9zdj krb5kdc[31727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:28.957298 31814 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7a8221339b4726842a82818d615874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33865 } }
I20260807 08:42:28.957515 31814 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:28.957556 31814 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7a8221339b4726842a82818d615874, State: Initialized, Role: FOLLOWER
I20260807 08:42:28.957744 31814 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7a8221339b4726842a82818d615874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33865 } }
I20260807 08:42:28.957821 31814 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:28.957883 31814 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:28.957935 31814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:28.958786 31814 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7a8221339b4726842a82818d615874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33865 } }
I20260807 08:42:28.958918 31814 leader_election.cc:302] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7a8221339b4726842a82818d615874; no voters:
I20260807 08:42:28.959115 31814 leader_election.cc:288] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:28.959188 31820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:28.959429 31820 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [term 1 LEADER]: Becoming Leader. State: Replica: ca7a8221339b4726842a82818d615874, State: Running, Role: LEADER
I20260807 08:42:28.959465 31814 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:28.959612 31820 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7a8221339b4726842a82818d615874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33865 } }
I20260807 08:42:28.960047 31822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca7a8221339b4726842a82818d615874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7a8221339b4726842a82818d615874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33865 } } }
I20260807 08:42:28.960150 31822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:28.961071 31821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca7a8221339b4726842a82818d615874. Latest consensus state: current_term: 1 leader_uuid: "ca7a8221339b4726842a82818d615874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7a8221339b4726842a82818d615874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33865 } } }
I20260807 08:42:28.961259 31821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:28.961428 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:28.947972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.28.108.190:33865)
0807 08:42:28.948201 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:28.948213 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:28.948258 (+ 45us) server_negotiation.cc:407] Connection header received
0807 08:42:28.949048 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:28.949060 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:28.949137 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:28.949324 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:28.950060 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:28.951632 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:28.952665 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:28.953410 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:28.953545 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:28.956688 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:28.956696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:28.956697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:28.956714 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:28.958990 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:28.959545 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:28.959549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:28.959549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:28.959677 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:28.959953 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:28.959955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:28.959955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:28.960418 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:28.960563 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:28.960795 (+ 232us) server_negotiation.cc:299] Negotiation successful
0807 08:42:28.961024 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
W20260807 08:42:28.961778 31835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:42:28.961864 31835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:42:28.962204 31829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:28.962432 31829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:28.964505 31829 catalog_manager.cc:1347] Generated new cluster ID: e670de7eb17040feaabda9478d56ab80
I20260807 08:42:28.964572 31829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:28.982654 31829 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:28.983459 31829 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:28.990499 31829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca7a8221339b4726842a82818d615874: Generated new TSK 0
I20260807 08:42:28.990792 31829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:42:29.038419 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33865
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:29.138422 31841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:29.138664 31841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:29.138690 31841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:29.140026 31841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:29.140084 31841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:29.140113 31841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:29.141912 31841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:29.142210 31841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:29.142503 31841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:29.148379 31846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:29.148458 31849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:29.148383 31847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:29.148619 31841 server_base.cc:1034] running on GCE node
I20260807 08:42:29.149109 31841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:29.149415 31841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:29.150635 31841 hybrid_clock.cc:648] HybridClock initialized: now 1786092149150578 us; error 80 us; skew 500 ppm
Aug 07 08:42:29 dist-test-slave-9zdj krb5kdc[31727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092149, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:29.154201 31841 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:42:29.154876 31841 webserver.cc:460] Webserver started at http://127.28.108.129:37963/ using document root <none> and password file <none>
I20260807 08:42:29.155151 31841 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:29.155225 31841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:29.155398 31841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:29.156534 31841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "4468b7605fcb4c68bbd2a9e581bcfeb1"
format_stamp: "Formatted at 2026-08-07 08:42:29 on dist-test-slave-9zdj"
server_key: "d56cf7a4fc0f5a92fbf2083c76b0ff85"
server_key_iv: "5f64d4db801f3922270e5337bd8a661b"
server_key_version: "encryptionkey@0"
I20260807 08:42:29.156926 31841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "4468b7605fcb4c68bbd2a9e581bcfeb1"
format_stamp: "Formatted at 2026-08-07 08:42:29 on dist-test-slave-9zdj"
server_key: "d56cf7a4fc0f5a92fbf2083c76b0ff85"
server_key_iv: "5f64d4db801f3922270e5337bd8a661b"
server_key_version: "encryptionkey@0"
I20260807 08:42:29.158529 31841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:42:29.159538 31856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:29.159746 31841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:29.159864 31841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "4468b7605fcb4c68bbd2a9e581bcfeb1"
format_stamp: "Formatted at 2026-08-07 08:42:29 on dist-test-slave-9zdj"
server_key: "d56cf7a4fc0f5a92fbf2083c76b0ff85"
server_key_iv: "5f64d4db801f3922270e5337bd8a661b"
server_key_version: "encryptionkey@0"
I20260807 08:42:29.159953 31841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:29.184504 31841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:29.187408 31841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:29.187549 31841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:29.187791 31841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:29.188217 31841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:29.188299 31841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:29.188341 31841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:29.188390 31841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:29.195474 31841 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:38081
I20260807 08:42:29.195513 31976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:38081 every 8 connection(s)
I20260807 08:42:29.195922 31841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
Aug 07 08:42:29 dist-test-slave-9zdj krb5kdc[31727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092149, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:29.206178 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 31841
I20260807 08:42:29.206300 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:29.206537 29106 external_mini_cluster.cc:1428] Setting key ff46dd8ed62570b8d1d822165c9ad5af
I20260807 08:42:29.209043 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:29.197388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:55305 (local address 127.28.108.190:33865)
0807 08:42:29.197494 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:29.197496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:29.197819 (+ 323us) server_negotiation.cc:407] Connection header received
0807 08:42:29.198459 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:29.198461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:29.198474 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:29.198548 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:29.199684 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.200324 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.201730 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.201963 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.202035 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:29.205180 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:29.205182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:29.205183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:29.205192 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:29.207380 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:29.207899 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:29.207900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:29.207901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:29.207970 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:29.208267 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:29.208268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:29.208268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:29.208409 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:29.208486 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:29.208820 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 08:42:29.208941 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 08:42:29.209280 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:29.197409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33865 (local address 127.28.108.129:55305)
0807 08:42:29.197593 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:29.197665 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:29.198190 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:29.198645 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:29.198649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:29.199340 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:29.199559 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:29.199564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.200586 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.200587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:29.201516 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:29.201520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.202084 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.202085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:29.202315 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:29.202965 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:29.202975 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:29.205026 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:29.207483 (+ 2457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:29.207497 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:29.207500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:29.207789 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:29.208046 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:29.208047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:29.208048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:29.208192 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:29.208488 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:29.208497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:29.208666 (+ 169us) client_negotiation.cc:769] Sending connection context
0807 08:42:29.208788 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 08:42:29.208989 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260807 08:42:29.209946 31977 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33865
I20260807 08:42:29.210125 31977 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:29.210381 31977 heartbeater.cc:511] Master 127.28.108.190:33865 requested a full tablet report, sending...
I20260807 08:42:29.211032 31776 ts_manager.cc:194] Registered new tserver with Master: 4468b7605fcb4c68bbd2a9e581bcfeb1 (127.28.108.129:38081)
I20260807 08:42:29.211771 31776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:55305
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:42:29.242523 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33865
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:29.331040 31982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:29.331293 31982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:29.331322 31982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:29.332597 31982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:29.332648 31982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:29.332677 31982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:29.334290 31982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:29.334590 31982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:29.334874 31982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:29.340631 31988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:29.340631 31991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:29.340631 31987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:29.341024 31982 server_base.cc:1034] running on GCE node
I20260807 08:42:29.341441 31982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:29.341797 31982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:29.342998 31982 hybrid_clock.cc:648] HybridClock initialized: now 1786092149342985 us; error 36 us; skew 500 ppm
Aug 07 08:42:29 dist-test-slave-9zdj krb5kdc[31727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092149, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:29.346140 31982 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:42:29.346915 31982 webserver.cc:460] Webserver started at http://127.28.108.130:43219/ using document root <none> and password file <none>
I20260807 08:42:29.347198 31982 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:29.347285 31982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:29.347483 31982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:29.348774 31982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "f61af4b2ee06470e8c1342b00d14e81c"
format_stamp: "Formatted at 2026-08-07 08:42:29 on dist-test-slave-9zdj"
server_key: "2e233f07bd00cee202dc506472472b79"
server_key_iv: "6653bca9e91a2e57707646fd60a8b64b"
server_key_version: "encryptionkey@0"
I20260807 08:42:29.349226 31982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "f61af4b2ee06470e8c1342b00d14e81c"
format_stamp: "Formatted at 2026-08-07 08:42:29 on dist-test-slave-9zdj"
server_key: "2e233f07bd00cee202dc506472472b79"
server_key_iv: "6653bca9e91a2e57707646fd60a8b64b"
server_key_version: "encryptionkey@0"
I20260807 08:42:29.351034 31982 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 08:42:29.352164 31999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:29.352607 31982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 08:42:29.352893 31982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "f61af4b2ee06470e8c1342b00d14e81c"
format_stamp: "Formatted at 2026-08-07 08:42:29 on dist-test-slave-9zdj"
server_key: "2e233f07bd00cee202dc506472472b79"
server_key_iv: "6653bca9e91a2e57707646fd60a8b64b"
server_key_version: "encryptionkey@0"
I20260807 08:42:29.353005 31982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:29.378564 31982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:29.381109 31982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:29.381325 31982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:29.381629 31982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:29.382000 31982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:29.382061 31982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:29.382098 31982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:29.382144 31982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:29.388392 31982 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:44383
I20260807 08:42:29.388444 32122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:44383 every 8 connection(s)
I20260807 08:42:29.388797 31982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:29.389914 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 31982
I20260807 08:42:29.390039 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:42:29.390255 29106 external_mini_cluster.cc:1428] Setting key 0409152d972ae4c828f67a4e586d0153
Aug 07 08:42:29 dist-test-slave-9zdj krb5kdc[31727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092149, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
I20260807 08:42:29.401467 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:29.389760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:41713 (local address 127.28.108.190:33865)
0807 08:42:29.389838 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:29.389841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:29.390059 (+ 218us) server_negotiation.cc:407] Connection header received
0807 08:42:29.390821 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:29.390823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:29.390837 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:29.390894 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:29.392509 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.393377 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.394562 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.394793 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.394889 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:29.398018 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:29.398020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:29.398021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:29.398030 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:29.399646 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:29.400153 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:29.400154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:29.400155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:29.400220 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:29.400617 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:29.400619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:29.400619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:29.400826 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:29.400897 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:29.401284 (+ 387us) server_negotiation.cc:299] Negotiation successful
0807 08:42:29.401389 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 08:42:29.401787 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:29.389669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33865 (local address 127.28.108.130:41713)
0807 08:42:29.389887 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:29.389946 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:29.390695 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:29.390978 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:29.390982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:29.391916 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:29.392387 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:29.392394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.393498 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.393500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:29.394461 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:29.394464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.394905 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.394906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:29.395046 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:29.395689 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:29.395698 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:29.397882 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:29.399763 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:29.399774 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:29.399777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:29.400055 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:29.400302 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:29.400303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:29.400303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:29.400453 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:29.400924 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:29.400932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:29.401086 (+ 154us) client_negotiation.cc:769] Sending connection context
0807 08:42:29.401227 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 08:42:29.401425 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
I20260807 08:42:29.402662 32123 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33865
I20260807 08:42:29.402911 32123 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:29.403154 32123 heartbeater.cc:511] Master 127.28.108.190:33865 requested a full tablet report, sending...
I20260807 08:42:29.403704 31776 ts_manager.cc:194] Registered new tserver with Master: f61af4b2ee06470e8c1342b00d14e81c (127.28.108.130:44383)
I20260807 08:42:29.404093 31776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:41713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:42:29.427102 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33865
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:29.521379 32128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:29.521711 32128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:29.521757 32128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:29.523329 32128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:29.523396 32128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:29.523452 32128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:29.525260 32128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:29.525511 32128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:29.525794 32128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:29.531898 32137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:29.531955 32135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:29.531975 32128 server_base.cc:1034] running on GCE node
W20260807 08:42:29.531906 32134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:29.532464 32128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:29.532827 32128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:29.534030 32128 hybrid_clock.cc:648] HybridClock initialized: now 1786092149533991 us; error 57 us; skew 500 ppm
Aug 07 08:42:29 dist-test-slave-9zdj krb5kdc[31727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092149, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:29.537200 32128 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:42:29.537837 32128 webserver.cc:460] Webserver started at http://127.28.108.131:33711/ using document root <none> and password file <none>
I20260807 08:42:29.538130 32128 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:29.538223 32128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:29.538419 32128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:29.539605 32128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "dc9954eb2aa546b9818c94963a4e1dbf"
format_stamp: "Formatted at 2026-08-07 08:42:29 on dist-test-slave-9zdj"
server_key: "d3d2a2c2ed5a6dc0cf18cebd5ad0d0d9"
server_key_iv: "ed24c4f69d4a83518d563ec7c09e41a2"
server_key_version: "encryptionkey@0"
I20260807 08:42:29.540021 32128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "dc9954eb2aa546b9818c94963a4e1dbf"
format_stamp: "Formatted at 2026-08-07 08:42:29 on dist-test-slave-9zdj"
server_key: "d3d2a2c2ed5a6dc0cf18cebd5ad0d0d9"
server_key_iv: "ed24c4f69d4a83518d563ec7c09e41a2"
server_key_version: "encryptionkey@0"
I20260807 08:42:29.541886 32128 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260807 08:42:29.543111 32144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:29.543579 32128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:29.543718 32128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "dc9954eb2aa546b9818c94963a4e1dbf"
format_stamp: "Formatted at 2026-08-07 08:42:29 on dist-test-slave-9zdj"
server_key: "d3d2a2c2ed5a6dc0cf18cebd5ad0d0d9"
server_key_iv: "ed24c4f69d4a83518d563ec7c09e41a2"
server_key_version: "encryptionkey@0"
I20260807 08:42:29.543879 32128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:29.555754 32128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:29.558925 32128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:29.559067 32128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:29.559362 32128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:29.559813 32128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:29.559862 32128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:29.559886 32128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:29.559899 32128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:29.568569 32128 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:39249
I20260807 08:42:29.568650 32264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:39249 every 8 connection(s)
I20260807 08:42:29.569074 32128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:29.574498 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 32128
I20260807 08:42:29.574638 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:29.574918 29106 external_mini_cluster.cc:1428] Setting key f9f888e8c77047eae532e49770fafaf3
Aug 07 08:42:29 dist-test-slave-9zdj krb5kdc[31727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092149, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:29.583659 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:29.570068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:43171 (local address 127.28.108.190:33865)
0807 08:42:29.570146 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:29.570149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:29.570421 (+ 272us) server_negotiation.cc:407] Connection header received
0807 08:42:29.571382 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:29.571383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:29.571396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:29.571460 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:29.573030 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.573795 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.575676 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.576009 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.576083 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:29.579783 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:29.579785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:29.579786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:29.579795 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:29.581564 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:29.582406 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:29.582408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:29.582408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:29.582480 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:29.582857 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:29.582859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:29.582859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:29.583009 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:29.583090 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:29.583485 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 08:42:29.583578 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 08:42:29.584028 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:29.570030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33865 (local address 127.28.108.131:43171)
0807 08:42:29.570243 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:29.570301 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:29.571219 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:29.571556 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:29.571561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:29.572497 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:29.572910 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:29.572919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.573915 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.573917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:29.575531 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:29.575535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:29.576130 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:29.576132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:29.576301 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:29.577155 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:29.577168 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:29.579624 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:29.581858 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:29.581871 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:29.581874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:29.582283 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:29.582565 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:29.582567 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:29.582567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:29.582749 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:29.583098 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:29.583106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:29.583322 (+ 216us) client_negotiation.cc:769] Sending connection context
0807 08:42:29.583456 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 08:42:29.583682 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":117,"threads_started":1}
I20260807 08:42:29.584801 32265 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33865
I20260807 08:42:29.584944 32265 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:29.585210 32265 heartbeater.cc:511] Master 127.28.108.190:33865 requested a full tablet report, sending...
I20260807 08:42:29.585745 31776 ts_manager.cc:194] Registered new tserver with Master: dc9954eb2aa546b9818c94963a4e1dbf (127.28.108.131:39249)
I20260807 08:42:29.586473 31776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:43171
I20260807 08:42:29.587248 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 08:42:29.603178 31818 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:29.601693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.28.108.190:33865)
0807 08:42:29.601772 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:29.601775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:29.602007 (+ 232us) server_negotiation.cc:407] Connection header received
0807 08:42:29.602044 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:29.602045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:29.602057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:29.602134 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:29.603062 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35562: BlockingRecv error: recv got EOF from 127.0.0.1:35562 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 08:42:29.604292 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 31841
I20260807 08:42:29.611536 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 31982
I20260807 08:42:29.619047 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 32128
I20260807 08:42:29.627079 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 31740
2026-08-07T08:42:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2158 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:29 dist-test-slave-9zdj krb5kdc[32277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:29 dist-test-slave-9zdj krb5kdc[32277](info): set up 2 sockets
Aug 07 08:42:29 dist-test-slave-9zdj krb5kdc[32277](info): commencing operation
krb5kdc: starting...
W20260807 08:42:30.753070 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:30 dist-test-slave-9zdj krb5kdc[32277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:30.907313 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42807
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:42807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:30.998245 32291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:30.998535 32291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:30.998577 32291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:30.999879 32291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:30.999969 32291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:30.999992 32291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:31.000015 32291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:31.000042 32291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:31.001950 32291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:42807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:31.002291 32291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:31.002564 32291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:31.009241 32296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:31.009413 32299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:31.009507 32297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:31.009423 32291 server_base.cc:1034] running on GCE node
I20260807 08:42:31.010030 32291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:31.010461 32291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:31.011709 32291 hybrid_clock.cc:648] HybridClock initialized: now 1786092151011661 us; error 67 us; skew 500 ppm
Aug 07 08:42:31 dist-test-slave-9zdj krb5kdc[32277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:31.019342 32291 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:31.019939 32291 webserver.cc:460] Webserver started at http://127.28.108.190:44491/ using document root <none> and password file <none>
I20260807 08:42:31.020208 32291 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:31.020283 32291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:31.020455 32291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:31.021790 32291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "b46d77ccca8745b2910055b2c8e3ad32"
format_stamp: "Formatted at 2026-08-07 08:42:31 on dist-test-slave-9zdj"
server_key: "535b01d148d6fb06a1a098af428651a4"
server_key_iv: "272bcda498df25e0abd128dd5725bdb9"
server_key_version: "encryptionkey@0"
I20260807 08:42:31.022183 32291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "b46d77ccca8745b2910055b2c8e3ad32"
format_stamp: "Formatted at 2026-08-07 08:42:31 on dist-test-slave-9zdj"
server_key: "535b01d148d6fb06a1a098af428651a4"
server_key_iv: "272bcda498df25e0abd128dd5725bdb9"
server_key_version: "encryptionkey@0"
I20260807 08:42:31.023797 32291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:31.024953 32307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:31.025281 32291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:31.025445 32291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "b46d77ccca8745b2910055b2c8e3ad32"
format_stamp: "Formatted at 2026-08-07 08:42:31 on dist-test-slave-9zdj"
server_key: "535b01d148d6fb06a1a098af428651a4"
server_key_iv: "272bcda498df25e0abd128dd5725bdb9"
server_key_version: "encryptionkey@0"
I20260807 08:42:31.025555 32291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:31.036316 32291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:31.038828 32291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:31.039050 32291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:31.043736 32291 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:42807
I20260807 08:42:31.043771 32366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:42807 every 8 connection(s)
I20260807 08:42:31.044420 32291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:31.044798 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 32291
I20260807 08:42:31.044893 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:31.045055 29106 external_mini_cluster.cc:1428] Setting key 79712bfb62fcd12c8b8ab28568ac7b8e
I20260807 08:42:31.045413 32367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:31.049180 32367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32: Bootstrap starting.
I20260807 08:42:31.050122 32367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:31.050498 32367 log.cc:824] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:31.051492 32367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32: No bootstrap required, opened a new log
Aug 07 08:42:31 dist-test-slave-9zdj krb5kdc[32277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:31.054776 32367 raft_consensus.cc:348] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46d77ccca8745b2910055b2c8e3ad32" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42807 } }
I20260807 08:42:31.055119 32367 raft_consensus.cc:374] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:31.055222 32367 raft_consensus.cc:729] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46d77ccca8745b2910055b2c8e3ad32, State: Initialized, Role: FOLLOWER
I20260807 08:42:31.055460 32367 consensus_queue.cc:260] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46d77ccca8745b2910055b2c8e3ad32" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42807 } }
I20260807 08:42:31.055670 32367 raft_consensus.cc:388] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:31.055773 32367 raft_consensus.cc:482] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:31.055891 32367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:31.056804 32367 raft_consensus.cc:504] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46d77ccca8745b2910055b2c8e3ad32" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42807 } }
I20260807 08:42:31.056939 32367 leader_election.cc:302] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b46d77ccca8745b2910055b2c8e3ad32; no voters:
I20260807 08:42:31.057160 32367 leader_election.cc:288] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:31.057379 32373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:31.057485 32367 sys_catalog.cc:564] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:31.057569 32373 raft_consensus.cc:686] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [term 1 LEADER]: Becoming Leader. State: Replica: b46d77ccca8745b2910055b2c8e3ad32, State: Running, Role: LEADER
I20260807 08:42:31.057729 32373 consensus_queue.cc:237] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46d77ccca8745b2910055b2c8e3ad32" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42807 } }
I20260807 08:42:31.058297 32373 sys_catalog.cc:455] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b46d77ccca8745b2910055b2c8e3ad32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46d77ccca8745b2910055b2c8e3ad32" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42807 } } }
I20260807 08:42:31.058394 32373 sys_catalog.cc:458] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:31.058537 32373 sys_catalog.cc:455] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b46d77ccca8745b2910055b2c8e3ad32. Latest consensus state: current_term: 1 leader_uuid: "b46d77ccca8745b2910055b2c8e3ad32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46d77ccca8745b2910055b2c8e3ad32" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42807 } } }
I20260807 08:42:31.058616 32373 sys_catalog.cc:458] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:31.059379 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:31.045739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.28.108.190:42807)
0807 08:42:31.045965 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:31.045975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:31.046013 (+ 38us) server_negotiation.cc:407] Connection header received
0807 08:42:31.046753 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:31.046765 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:31.046842 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:31.047035 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:31.047791 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:31.049215 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:31.050256 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:31.051059 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:31.051173 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:31.054825 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:31.054831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:31.054831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:31.054849 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:31.057152 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:31.057715 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:31.057719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:31.057720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:31.057884 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:31.058109 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:31.058110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:31.058111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:31.058499 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:31.058650 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:31.058963 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 08:42:31.059129 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":97,"threads_started":1}
W20260807 08:42:31.059617 32387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:42:31.060122 32387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:42:31.060617 32381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:31.060726 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:42:31.061896 32381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:31.063601 32389 mvcc.cc:205] Tried to move back new op lower bound from 7315833450754142208 to 7315833450733223936. Current Snapshot: MvccSnapshot[applied={T|T < 7315833450754142208}]
I20260807 08:42:31.064102 32381 catalog_manager.cc:1347] Generated new cluster ID: 2f9e2a946d394d7baae9b92ae5df3f69
I20260807 08:42:31.064165 32381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:31.087030 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:31.082265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.28.108.190:42807)
0807 08:42:31.082313 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:31.082316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:31.082504 (+ 188us) server_negotiation.cc:407] Connection header received
0807 08:42:31.082562 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:31.082563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:31.082580 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:31.082641 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:31.083133 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:31.083941 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:31.085065 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:31.085460 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:31.085546 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:31.086439 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:31.086441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:31.086442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:31.086721 (+ 279us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:31.086746 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:31.086754 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:31.086823 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:31.086894 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 08:42:31.086930 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:42:31.089360 32381 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:31.090195 32381 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:31.095973 32381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b46d77ccca8745b2910055b2c8e3ad32: Generated new TSK 0
I20260807 08:42:31.096277 32381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:42:31.101584 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 32291
2026-08-07T08:42:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1473 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:31 dist-test-slave-9zdj krb5kdc[32401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:31 dist-test-slave-9zdj krb5kdc[32401](info): set up 2 sockets
Aug 07 08:42:31 dist-test-slave-9zdj krb5kdc[32401](info): commencing operation
krb5kdc: starting...
W20260807 08:42:32.272145 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:32 dist-test-slave-9zdj krb5kdc[32401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:32.427179 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:38167
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:40523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:38167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:32.523931 32414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:32.524227 32414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:32.524282 32414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:32.526600 32414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:32.526718 32414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:32.526746 32414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:32.526770 32414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:32.526803 32414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:32.528870 32414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:40523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:38167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:38167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:32.529274 32414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:32.529600 32414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:42:32.536046 32414 server_base.cc:1034] running on GCE node
W20260807 08:42:32.536053 32420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:32.536047 32419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:32.536453 32422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:32.536839 32414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:32.537207 32414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:32.538414 32414 hybrid_clock.cc:648] HybridClock initialized: now 1786092152538396 us; error 53 us; skew 500 ppm
Aug 07 08:42:32 dist-test-slave-9zdj krb5kdc[32401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:32.542325 32414 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:32.543111 32414 webserver.cc:460] Webserver started at http://127.28.108.190:33861/ using document root <none> and password file <none>
I20260807 08:42:32.543396 32414 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:32.543450 32414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:32.543602 32414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:32.544824 32414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "24d11e1d8d3343ff98890bb67c8f8bdf"
format_stamp: "Formatted at 2026-08-07 08:42:32 on dist-test-slave-9zdj"
server_key: "06f7e7f1e262f65df161a5b7869a3236"
server_key_iv: "01ecb168bad7f4dccd45ff92464d5756"
server_key_version: "encryptionkey@0"
I20260807 08:42:32.545292 32414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "24d11e1d8d3343ff98890bb67c8f8bdf"
format_stamp: "Formatted at 2026-08-07 08:42:32 on dist-test-slave-9zdj"
server_key: "06f7e7f1e262f65df161a5b7869a3236"
server_key_iv: "01ecb168bad7f4dccd45ff92464d5756"
server_key_version: "encryptionkey@0"
I20260807 08:42:32.546970 32414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 08:42:32.547941 32430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:32.548148 32414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:32.548265 32414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "24d11e1d8d3343ff98890bb67c8f8bdf"
format_stamp: "Formatted at 2026-08-07 08:42:32 on dist-test-slave-9zdj"
server_key: "06f7e7f1e262f65df161a5b7869a3236"
server_key_iv: "01ecb168bad7f4dccd45ff92464d5756"
server_key_version: "encryptionkey@0"
I20260807 08:42:32.548343 32414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:32.556635 32414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:32.559409 32414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:32.559638 32414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:32.565526 32414 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:38167
I20260807 08:42:32.565572 32487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:38167 every 8 connection(s)
I20260807 08:42:32.566078 32414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:32.567413 32489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:32.570353 32489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf: Bootstrap starting.
I20260807 08:42:32.571074 32489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:32.571444 32489 log.cc:824] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:32.572485 32489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf: No bootstrap required, opened a new log
I20260807 08:42:32.574481 32489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d11e1d8d3343ff98890bb67c8f8bdf" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 38167 } }
I20260807 08:42:32.574748 32489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:32.574800 32489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24d11e1d8d3343ff98890bb67c8f8bdf, State: Initialized, Role: FOLLOWER
I20260807 08:42:32.574899 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 32414
I20260807 08:42:32.575013 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:32.574990 32489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d11e1d8d3343ff98890bb67c8f8bdf" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 38167 } }
I20260807 08:42:32.575085 32489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:32.575160 32489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:32.575237 29106 external_mini_cluster.cc:1428] Setting key 2cddcddbc848dc77db4b8f9dacb0181c
I20260807 08:42:32.575230 32489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:32.576488 32489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d11e1d8d3343ff98890bb67c8f8bdf" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 38167 } }
I20260807 08:42:32.576622 32489 leader_election.cc:302] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24d11e1d8d3343ff98890bb67c8f8bdf; no voters:
I20260807 08:42:32.576824 32489 leader_election.cc:288] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:32.576844 32494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:32.576956 32494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [term 1 LEADER]: Becoming Leader. State: Replica: 24d11e1d8d3343ff98890bb67c8f8bdf, State: Running, Role: LEADER
I20260807 08:42:32.577062 32494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d11e1d8d3343ff98890bb67c8f8bdf" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 38167 } }
I20260807 08:42:32.577129 32489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:32.577462 32494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24d11e1d8d3343ff98890bb67c8f8bdf. Latest consensus state: current_term: 1 leader_uuid: "24d11e1d8d3343ff98890bb67c8f8bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d11e1d8d3343ff98890bb67c8f8bdf" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 38167 } } }
I20260807 08:42:32.577557 32494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:32.577476 32496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24d11e1d8d3343ff98890bb67c8f8bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d11e1d8d3343ff98890bb67c8f8bdf" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 38167 } } }
I20260807 08:42:32.577785 32496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:32.577874 32500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:32.578500 32500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:32.580519 32500 catalog_manager.cc:1347] Generated new cluster ID: 6a3504ed0f2c438790c3e6a706acd203
I20260807 08:42:32.580577 32500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:42:32 dist-test-slave-9zdj krb5kdc[32401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:32.588973 32500 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:32.589576 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:32.575816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.28.108.190:38167)
0807 08:42:32.576065 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:32.576080 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:32.576123 (+ 43us) server_negotiation.cc:407] Connection header received
0807 08:42:32.576259 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:32.576267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:32.576331 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:32.576576 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:32.577685 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:32.579132 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:32.580355 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:32.581019 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:32.581133 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:32.584393 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:32.584399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:32.584400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:32.584417 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:32.587130 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:32.587686 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:32.587691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:32.587692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:32.587890 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:32.588168 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:32.588169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:32.588169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:32.588592 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:32.588918 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:32.589025 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 08:42:32.589301 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":116,"threads_started":1}
I20260807 08:42:32.589690 32500 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:32.592579 32500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24d11e1d8d3343ff98890bb67c8f8bdf: Generated new TSK 0
I20260807 08:42:32.592840 32500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:42:32.629338 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:38167
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:40523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:32.744590 32517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:32.744845 32517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:32.744875 32517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:32.746428 32517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:32.746492 32517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:32.746526 32517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:32.748462 32517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:40523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:38167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:32.748797 32517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:32.749121 32517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:32.755318 32525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:32.755494 32522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:32.755511 32523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:32.755661 32517 server_base.cc:1034] running on GCE node
I20260807 08:42:32.755869 32517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:32.756100 32517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:32.757298 32517 hybrid_clock.cc:648] HybridClock initialized: now 1786092152757277 us; error 45 us; skew 500 ppm
Aug 07 08:42:32 dist-test-slave-9zdj krb5kdc[32401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:32.760322 32517 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:42:32.760850 32517 webserver.cc:460] Webserver started at http://127.28.108.129:43205/ using document root <none> and password file <none>
I20260807 08:42:32.761087 32517 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:32.761137 32517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:32.761240 32517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:32.762452 32517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "41e3839aec51418bb02dd3de3ec3dc97"
format_stamp: "Formatted at 2026-08-07 08:42:32 on dist-test-slave-9zdj"
server_key: "53f2b7a46d681d042a2863cbdb35c498"
server_key_iv: "c2fdc9ce1e69cb61781bd745b09eef5e"
server_key_version: "encryptionkey@0"
I20260807 08:42:32.762812 32517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "41e3839aec51418bb02dd3de3ec3dc97"
format_stamp: "Formatted at 2026-08-07 08:42:32 on dist-test-slave-9zdj"
server_key: "53f2b7a46d681d042a2863cbdb35c498"
server_key_iv: "c2fdc9ce1e69cb61781bd745b09eef5e"
server_key_version: "encryptionkey@0"
I20260807 08:42:32.764384 32517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:42:32.765407 32532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:32.765702 32517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:32.765884 32517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "41e3839aec51418bb02dd3de3ec3dc97"
format_stamp: "Formatted at 2026-08-07 08:42:32 on dist-test-slave-9zdj"
server_key: "53f2b7a46d681d042a2863cbdb35c498"
server_key_iv: "c2fdc9ce1e69cb61781bd745b09eef5e"
server_key_version: "encryptionkey@0"
I20260807 08:42:32.765982 32517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:32.777777 32517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:32.780750 32517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:32.780978 32517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:32.781272 32517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:32.781685 32517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:32.781750 32517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:32.781795 32517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:32.781837 32517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:32.788501 32517 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:45081
I20260807 08:42:32.788550 32652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:45081 every 8 connection(s)
I20260807 08:42:32.788978 32517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
Aug 07 08:42:32 dist-test-slave-9zdj krb5kdc[32401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:32.798368 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 32517
I20260807 08:42:32.798527 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:32.798774 29106 external_mini_cluster.cc:1428] Setting key 79d89d8e4742372e000249e1f11feeb2
I20260807 08:42:32.800925 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:32.790008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:33341 (local address 127.28.108.190:38167)
0807 08:42:32.790084 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:32.790086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:32.790400 (+ 314us) server_negotiation.cc:407] Connection header received
0807 08:42:32.790837 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:32.790839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:32.790858 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:32.790927 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:32.792128 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:32.792709 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:32.794160 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:32.794376 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:32.794438 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:32.797052 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:32.797054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:32.797054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:32.797063 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:32.798981 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:32.799614 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:32.799615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:32.799616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:32.799694 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:32.799993 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:32.799994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:32.799995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:32.800165 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:32.800241 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:32.800676 (+ 435us) server_negotiation.cc:299] Negotiation successful
0807 08:42:32.800788 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 08:42:32.801154 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:32.789961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:38167 (local address 127.28.108.129:33341)
0807 08:42:32.790155 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:32.790207 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:32.790733 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:32.791011 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:32.791014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:32.791778 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:32.792003 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:32.792009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:32.792826 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:32.792828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:32.794028 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:32.794032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:32.794477 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:32.794478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:32.794613 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:32.795207 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:32.795218 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:32.796926 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:32.799094 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:32.799106 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:32.799109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:32.799513 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:32.799772 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:32.799773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:32.799773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:32.799905 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:32.800260 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:32.800267 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:32.800511 (+ 244us) client_negotiation.cc:769] Sending connection context
0807 08:42:32.800631 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 08:42:32.800790 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":115,"threads_started":1}
I20260807 08:42:32.801950 32653 heartbeater.cc:347] Connected to a master server at 127.28.108.190:38167
I20260807 08:42:32.802074 32653 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:32.802285 32653 heartbeater.cc:511] Master 127.28.108.190:38167 requested a full tablet report, sending...
I20260807 08:42:32.803004 32449 ts_manager.cc:194] Registered new tserver with Master: 41e3839aec51418bb02dd3de3ec3dc97 (127.28.108.129:45081)
I20260807 08:42:32.803757 32449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:33341
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:42:32.838421 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:38167
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:40523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:32.930135 32658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:32.930372 32658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:32.930399 32658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:32.931576 32658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:32.931617 32658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:32.931645 32658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:32.933159 32658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:40523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:38167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:32.933383 32658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:32.933585 32658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:32.940691 32666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:32.940709 32667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:32.940866 32669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:32.940730 32658 server_base.cc:1034] running on GCE node
I20260807 08:42:32.941217 32658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:32.941584 32658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:32.942830 32658 hybrid_clock.cc:648] HybridClock initialized: now 1786092152942812 us; error 37 us; skew 500 ppm
Aug 07 08:42:32 dist-test-slave-9zdj krb5kdc[32401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:32.946178 32658 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:42:32.946774 32658 webserver.cc:460] Webserver started at http://127.28.108.130:44975/ using document root <none> and password file <none>
I20260807 08:42:32.947079 32658 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:32.947160 32658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:32.947319 32658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:32.948506 32658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "a0d9e153f58f4d769a1ba431a522b725"
format_stamp: "Formatted at 2026-08-07 08:42:32 on dist-test-slave-9zdj"
server_key: "b5e1ccd9b73bbcc2d7a6c18ba857fbce"
server_key_iv: "a69c017fc4221b895b15444133986a99"
server_key_version: "encryptionkey@0"
I20260807 08:42:32.948915 32658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "a0d9e153f58f4d769a1ba431a522b725"
format_stamp: "Formatted at 2026-08-07 08:42:32 on dist-test-slave-9zdj"
server_key: "b5e1ccd9b73bbcc2d7a6c18ba857fbce"
server_key_iv: "a69c017fc4221b895b15444133986a99"
server_key_version: "encryptionkey@0"
I20260807 08:42:32.950583 32658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:42:32.951535 32676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:32.951756 32658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:32.951885 32658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "a0d9e153f58f4d769a1ba431a522b725"
format_stamp: "Formatted at 2026-08-07 08:42:32 on dist-test-slave-9zdj"
server_key: "b5e1ccd9b73bbcc2d7a6c18ba857fbce"
server_key_iv: "a69c017fc4221b895b15444133986a99"
server_key_version: "encryptionkey@0"
I20260807 08:42:32.951987 32658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:32.969431 32658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:32.972573 32658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:32.972862 32658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:32.973188 32658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:32.973567 32658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:32.973661 32658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:32.973703 32658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:32.973753 32658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:32.981690 32658 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:38281
I20260807 08:42:32.981768 330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:38281 every 8 connection(s)
I20260807 08:42:32.982229 32658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:32.986179 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 32658
I20260807 08:42:32.986297 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:42:32.986610 29106 external_mini_cluster.cc:1428] Setting key 9fcbe6f39d1196e8fd8ceba1827dd1e4
Aug 07 08:42:32 dist-test-slave-9zdj krb5kdc[32401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:32.995273 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:32.983122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:41239 (local address 127.28.108.190:38167)
0807 08:42:32.983213 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:32.983216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:32.983538 (+ 322us) server_negotiation.cc:407] Connection header received
0807 08:42:32.984113 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:32.984114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:32.984128 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:32.984201 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:32.985348 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:32.986020 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:32.987744 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:32.987976 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:32.988080 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:32.991358 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:32.991361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:32.991361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:32.991370 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:32.993123 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:32.993827 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:32.993828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:32.993829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:32.993929 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:32.994335 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:32.994336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:32.994337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:32.994583 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:32.994650 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:32.995108 (+ 458us) server_negotiation.cc:299] Negotiation successful
0807 08:42:32.995195 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 08:42:32.995653 333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:32.983100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:38167 (local address 127.28.108.130:41239)
0807 08:42:32.983325 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:32.983440 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:32.983996 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:32.984261 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:32.984265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:32.984953 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:32.985202 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:32.985208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:32.986153 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:32.986155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:32.987641 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:32.987644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:32.988095 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:32.988096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:32.988370 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:32.989101 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:32.989117 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:32.991231 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:32.993244 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:32.993256 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:32.993259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:32.993711 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:32.994017 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:32.994018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:32.994018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:32.994236 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:32.994670 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:32.994680 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:32.994922 (+ 242us) client_negotiation.cc:769] Sending connection context
0807 08:42:32.995091 (+ 169us) client_negotiation.cc:240] Negotiation successful
0807 08:42:32.995340 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":131,"threads_started":1}
I20260807 08:42:32.996291 331 heartbeater.cc:347] Connected to a master server at 127.28.108.190:38167
I20260807 08:42:32.996435 331 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:32.996678 331 heartbeater.cc:511] Master 127.28.108.190:38167 requested a full tablet report, sending...
I20260807 08:42:32.997246 32449 ts_manager.cc:194] Registered new tserver with Master: a0d9e153f58f4d769a1ba431a522b725 (127.28.108.130:38281)
I20260807 08:42:32.997787 32449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:41239
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:42:33.021843 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:38167
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:40523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:33.119520 336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:33.119786 336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:33.119819 336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:33.121050 336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:33.121099 336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:33.121129 336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:33.122821 336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:40523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:38167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:33.123062 336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:33.123283 336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:33.129072 345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:33.129097 342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:33.129097 343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:33.129747 336 server_base.cc:1034] running on GCE node
I20260807 08:42:33.129993 336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:33.130235 336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:33.131430 336 hybrid_clock.cc:648] HybridClock initialized: now 1786092153131398 us; error 46 us; skew 500 ppm
Aug 07 08:42:33 dist-test-slave-9zdj krb5kdc[32401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:33.134562 336 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:42:33.135149 336 webserver.cc:460] Webserver started at http://127.28.108.131:43797/ using document root <none> and password file <none>
I20260807 08:42:33.135428 336 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:33.135483 336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:33.135586 336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:33.136742 336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "5766fb2af5954d72a66ef2029de00748"
format_stamp: "Formatted at 2026-08-07 08:42:33 on dist-test-slave-9zdj"
server_key: "bb52a76d3d9f0b4092b655e9984d480f"
server_key_iv: "d700c63e1170138116a6f78c4a7889cc"
server_key_version: "encryptionkey@0"
I20260807 08:42:33.137063 336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "5766fb2af5954d72a66ef2029de00748"
format_stamp: "Formatted at 2026-08-07 08:42:33 on dist-test-slave-9zdj"
server_key: "bb52a76d3d9f0b4092b655e9984d480f"
server_key_iv: "d700c63e1170138116a6f78c4a7889cc"
server_key_version: "encryptionkey@0"
I20260807 08:42:33.138595 336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260807 08:42:33.139585 353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:33.139910 336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:33.140002 336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "5766fb2af5954d72a66ef2029de00748"
format_stamp: "Formatted at 2026-08-07 08:42:33 on dist-test-slave-9zdj"
server_key: "bb52a76d3d9f0b4092b655e9984d480f"
server_key_iv: "d700c63e1170138116a6f78c4a7889cc"
server_key_version: "encryptionkey@0"
I20260807 08:42:33.140196 336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:33.157169 336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:33.160017 336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:33.160200 336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:33.160511 336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:33.160890 336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:33.160931 336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:33.160957 336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:33.160970 336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:33.167889 336 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:34661
I20260807 08:42:33.167938 471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:34661 every 8 connection(s)
I20260807 08:42:33.168538 336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:33.169445 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 336
I20260807 08:42:33.169548 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:33.169883 29106 external_mini_cluster.cc:1428] Setting key 91788d4717b5216ab89c7fc3b2676225
Aug 07 08:42:33 dist-test-slave-9zdj krb5kdc[32401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:33.180243 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:33.169530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:54237 (local address 127.28.108.190:38167)
0807 08:42:33.169707 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:33.169710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:33.169914 (+ 204us) server_negotiation.cc:407] Connection header received
0807 08:42:33.170674 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:33.170679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:33.170693 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:33.170771 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:33.171996 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:33.172560 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:33.173819 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:33.174072 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:33.174143 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:33.176826 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:33.176828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:33.176829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:33.176837 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:33.178696 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:33.179229 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:33.179230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:33.179231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:33.179296 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:33.179574 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:33.179575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:33.179575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:33.179715 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:33.179773 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:33.180088 (+ 315us) server_negotiation.cc:299] Negotiation successful
0807 08:42:33.180171 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260807 08:42:33.180500 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:33.169491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:38167 (local address 127.28.108.131:54237)
0807 08:42:33.169731 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:33.169774 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:33.170522 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:33.170918 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:33.170924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:33.171645 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:33.171878 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:33.171883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:33.172683 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:33.172683 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:33.173702 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:33.173705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:33.174170 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:33.174171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:33.174296 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:33.174986 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:33.174996 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:33.176696 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:33.178822 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:33.178859 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:33.178861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:33.179142 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:33.179370 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:33.179371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:33.179372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:33.179508 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:33.179783 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:33.179788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:33.179961 (+ 173us) client_negotiation.cc:769] Sending connection context
0807 08:42:33.180056 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 08:42:33.180213 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":221,"threads_started":1}
I20260807 08:42:33.181097 472 heartbeater.cc:347] Connected to a master server at 127.28.108.190:38167
I20260807 08:42:33.181219 472 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:33.181481 472 heartbeater.cc:511] Master 127.28.108.190:38167 requested a full tablet report, sending...
I20260807 08:42:33.181965 32449 ts_manager.cc:194] Registered new tserver with Master: 5766fb2af5954d72a66ef2029de00748 (127.28.108.131:34661)
I20260807 08:42:33.182214 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:42:33.182528 32449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:54237
Aug 07 08:42:33 dist-test-slave-9zdj krb5kdc[32401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092153, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 08:42:33 dist-test-slave-9zdj krb5kdc[32401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092153, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:33.211537 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:33.202195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.28.108.190:38167)
0807 08:42:33.202312 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:33.202314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:33.202421 (+ 107us) server_negotiation.cc:407] Connection header received
0807 08:42:33.202461 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:33.202462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:33.202474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:33.202538 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:33.203247 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:33.203789 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:33.204614 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:33.204869 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:33.204936 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:33.207909 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:33.207911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:33.207912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:33.207922 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:33.209888 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:33.210317 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:33.210319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:33.210319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:33.210404 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:33.210694 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:33.210695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:33.210696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:33.210891 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:33.210988 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:33.211166 (+ 178us) server_negotiation.cc:299] Negotiation successful
0807 08:42:33.211443 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260807 08:42:33.211874 32449 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41098
I20260807 08:42:33.212634 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 32517
I20260807 08:42:33.219897 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 32658
I20260807 08:42:33.228078 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 336
I20260807 08:42:33.239064 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 32414
2026-08-07T08:42:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2137 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:33 dist-test-slave-9zdj krb5kdc[486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:33 dist-test-slave-9zdj krb5kdc[486](info): set up 2 sockets
Aug 07 08:42:33 dist-test-slave-9zdj krb5kdc[486](info): commencing operation
krb5kdc: starting...
W20260807 08:42:34.395818 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:34 dist-test-slave-9zdj krb5kdc[486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:34.551131 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:44307
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:44307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:34.641002 504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:34.641260 504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:34.641286 504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:34.642655 504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:34.642721 504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:34.642737 504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:34.642751 504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:34.642764 504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:34.644522 504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:44307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:44307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:34.644846 504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:34.645155 504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:34.651299 512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:34.651299 509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:34.651325 510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:34.651928 504 server_base.cc:1034] running on GCE node
I20260807 08:42:34.652196 504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:34.652504 504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:34.653736 504 hybrid_clock.cc:648] HybridClock initialized: now 1786092154653729 us; error 44 us; skew 500 ppm
Aug 07 08:42:34 dist-test-slave-9zdj krb5kdc[486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:34.656819 504 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:34.657385 504 webserver.cc:460] Webserver started at http://127.28.108.190:34623/ using document root <none> and password file <none>
I20260807 08:42:34.657744 504 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:34.657835 504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:34.658056 504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:34.659406 504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "59524fcf3c42424a97061884a85f8b51"
format_stamp: "Formatted at 2026-08-07 08:42:34 on dist-test-slave-9zdj"
server_key: "5e4eae4dbc530a729bc940b8d08ca824"
server_key_iv: "4e851c7416b700e1f99376891c5f1103"
server_key_version: "encryptionkey@0"
I20260807 08:42:34.659847 504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "59524fcf3c42424a97061884a85f8b51"
format_stamp: "Formatted at 2026-08-07 08:42:34 on dist-test-slave-9zdj"
server_key: "5e4eae4dbc530a729bc940b8d08ca824"
server_key_iv: "4e851c7416b700e1f99376891c5f1103"
server_key_version: "encryptionkey@0"
I20260807 08:42:34.661554 504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 08:42:34.662667 520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:34.662971 504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:34.663127 504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "59524fcf3c42424a97061884a85f8b51"
format_stamp: "Formatted at 2026-08-07 08:42:34 on dist-test-slave-9zdj"
server_key: "5e4eae4dbc530a729bc940b8d08ca824"
server_key_iv: "4e851c7416b700e1f99376891c5f1103"
server_key_version: "encryptionkey@0"
I20260807 08:42:34.663228 504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:34.690148 504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:34.692883 504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:34.693092 504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:34.697831 504 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:44307
I20260807 08:42:34.697870 576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:44307 every 8 connection(s)
I20260807 08:42:34.698457 504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:34.699242 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 504
I20260807 08:42:34.699349 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:34.699577 29106 external_mini_cluster.cc:1428] Setting key 7464846796792058b1e36a92faa6820e
I20260807 08:42:34.699572 577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:34.703738 577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51: Bootstrap starting.
I20260807 08:42:34.704581 577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:34.704952 577 log.cc:824] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:34.706144 577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51: No bootstrap required, opened a new log
Aug 07 08:42:34 dist-test-slave-9zdj krb5kdc[486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:34.709000 577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59524fcf3c42424a97061884a85f8b51" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44307 } }
I20260807 08:42:34.709185 577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:34.709211 577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59524fcf3c42424a97061884a85f8b51, State: Initialized, Role: FOLLOWER
I20260807 08:42:34.709394 577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59524fcf3c42424a97061884a85f8b51" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44307 } }
I20260807 08:42:34.709506 577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:34.709545 577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:34.709595 577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:34.710325 577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59524fcf3c42424a97061884a85f8b51" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44307 } }
I20260807 08:42:34.710429 577 leader_election.cc:302] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59524fcf3c42424a97061884a85f8b51; no voters:
I20260807 08:42:34.710707 577 leader_election.cc:288] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:34.710741 582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:34.710875 582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [term 1 LEADER]: Becoming Leader. State: Replica: 59524fcf3c42424a97061884a85f8b51, State: Running, Role: LEADER
I20260807 08:42:34.710979 582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59524fcf3c42424a97061884a85f8b51" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44307 } }
I20260807 08:42:34.711045 577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:34.711441 583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59524fcf3c42424a97061884a85f8b51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59524fcf3c42424a97061884a85f8b51" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44307 } } }
I20260807 08:42:34.711632 583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:34.712169 589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:34.712852 589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:34.712937 584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59524fcf3c42424a97061884a85f8b51. Latest consensus state: current_term: 1 leader_uuid: "59524fcf3c42424a97061884a85f8b51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59524fcf3c42424a97061884a85f8b51" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44307 } } }
I20260807 08:42:34.713100 584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:34.713467 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:34.700284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37220 (local address 127.28.108.190:44307)
0807 08:42:34.700494 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:34.700502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:34.700538 (+ 36us) server_negotiation.cc:407] Connection header received
0807 08:42:34.700999 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:34.701008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:34.701061 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:34.701196 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:34.702152 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:34.703211 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:34.704333 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:34.704844 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:34.704952 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:34.708391 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:34.708396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:34.708396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:34.708410 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:34.710933 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:34.711514 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:34.711518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:34.711519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:34.711650 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:34.711949 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:34.711949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:34.711950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:34.712322 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:34.712497 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:34.713063 (+ 566us) server_negotiation.cc:299] Negotiation successful
0807 08:42:34.713232 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260807 08:42:34.714787 589 catalog_manager.cc:1347] Generated new cluster ID: 304d81d3347d4d449d2602324ebe662f
I20260807 08:42:34.714850 589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:34.735888 589 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:34.736683 589 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:34.753538 589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59524fcf3c42424a97061884a85f8b51: Generated new TSK 0
I20260807 08:42:34.753887 589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:42:34.790776 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:44307
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:34.892426 605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:34.892675 605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:34.892704 605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:34.894022 605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:34.894073 605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:34.894104 605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:34.895715 605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:44307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:34.895942 605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:34.896144 605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:34.902408 611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:34.902506 610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:34.902446 613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:34.902668 605 server_base.cc:1034] running on GCE node
I20260807 08:42:34.903014 605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:34.903321 605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:34.904524 605 hybrid_clock.cc:648] HybridClock initialized: now 1786092154904509 us; error 32 us; skew 500 ppm
Aug 07 08:42:34 dist-test-slave-9zdj krb5kdc[486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:34.907828 605 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:42:34.908387 605 webserver.cc:460] Webserver started at http://127.28.108.129:35959/ using document root <none> and password file <none>
I20260807 08:42:34.908686 605 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:34.908769 605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:34.908950 605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:34.910363 605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "5c83635f9be049fbb5ffcac98ff3f5bc"
format_stamp: "Formatted at 2026-08-07 08:42:34 on dist-test-slave-9zdj"
server_key: "1523eaf5a221e9905a9189aa24e4c5aa"
server_key_iv: "72227f2a311be2f273f4dac8560b7311"
server_key_version: "encryptionkey@0"
I20260807 08:42:34.910782 605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "5c83635f9be049fbb5ffcac98ff3f5bc"
format_stamp: "Formatted at 2026-08-07 08:42:34 on dist-test-slave-9zdj"
server_key: "1523eaf5a221e9905a9189aa24e4c5aa"
server_key_iv: "72227f2a311be2f273f4dac8560b7311"
server_key_version: "encryptionkey@0"
I20260807 08:42:34.912657 605 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 08:42:34.913728 621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:34.913944 605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:34.914062 605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "5c83635f9be049fbb5ffcac98ff3f5bc"
format_stamp: "Formatted at 2026-08-07 08:42:34 on dist-test-slave-9zdj"
server_key: "1523eaf5a221e9905a9189aa24e4c5aa"
server_key_iv: "72227f2a311be2f273f4dac8560b7311"
server_key_version: "encryptionkey@0"
I20260807 08:42:34.914165 605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:34.944603 605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:34.947657 605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:34.947872 605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:34.948176 605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:34.948622 605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:34.948684 605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:34.948722 605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:34.948757 605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:34.955535 605 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:33967
I20260807 08:42:34.955626 744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:33967 every 8 connection(s)
I20260807 08:42:34.956051 605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:42:34.958792 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 605
I20260807 08:42:34.959010 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:34.959233 29106 external_mini_cluster.cc:1428] Setting key 3f09c0df880bc3ba70bba3800eceef80
Aug 07 08:42:34 dist-test-slave-9zdj krb5kdc[486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:34.971282 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:34.956986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:57489 (local address 127.28.108.190:44307)
0807 08:42:34.957060 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:34.957062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:34.957357 (+ 295us) server_negotiation.cc:407] Connection header received
0807 08:42:34.958486 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:34.958488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:34.958501 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:34.958572 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:34.959662 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:34.960294 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:34.962189 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:34.962664 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:34.962759 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:34.966490 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:34.966492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:34.966493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:34.966505 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:34.968829 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:34.969850 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:34.969852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:34.969853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:34.969951 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:34.970354 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:34.970355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:34.970356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:34.970545 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:34.970630 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:34.971026 (+ 396us) server_negotiation.cc:299] Negotiation successful
0807 08:42:34.971146 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 08:42:34.971308 747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:34.956972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:44307 (local address 127.28.108.129:57489)
0807 08:42:34.957171 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:34.957223 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:34.958303 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:34.958714 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:34.958720 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:34.959194 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:34.959571 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:34.959578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:34.960410 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:34.960412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:34.962031 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:34.962036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:34.962788 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:34.962789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:34.962952 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:34.963806 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:34.963818 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:34.966344 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:34.969157 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:34.969172 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:34.969175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:34.969717 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:34.970071 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:34.970073 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:34.970074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:34.970242 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:34.970637 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:34.970645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:34.970825 (+ 180us) client_negotiation.cc:769] Sending connection context
0807 08:42:34.970935 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 08:42:34.971108 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260807 08:42:34.972043 745 heartbeater.cc:347] Connected to a master server at 127.28.108.190:44307
I20260807 08:42:34.972198 745 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
I20260807 08:42:34.972486 745 heartbeater.cc:511] Master 127.28.108.190:44307 requested a full tablet report, sending...
I20260807 08:42:34.973445 539 ts_manager.cc:194] Registered new tserver with Master: 5c83635f9be049fbb5ffcac98ff3f5bc (127.28.108.129:33967)
I20260807 08:42:34.974359 539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:57489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:42:34.999230 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:44307
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:35.096462 750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:35.096732 750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:35.096762 750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:35.098227 750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:35.098294 750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:35.098325 750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:35.100152 750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:44307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:35.100453 750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:35.100757 750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:35.106977 755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:35.106977 758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:35.106984 756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:35.107403 750 server_base.cc:1034] running on GCE node
I20260807 08:42:35.107766 750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:35.108143 750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:35.109467 750 hybrid_clock.cc:648] HybridClock initialized: now 1786092155109451 us; error 38 us; skew 500 ppm
Aug 07 08:42:35 dist-test-slave-9zdj krb5kdc[486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092155, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:35.112871 750 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:42:35.113410 750 webserver.cc:460] Webserver started at http://127.28.108.130:43731/ using document root <none> and password file <none>
I20260807 08:42:35.113729 750 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:35.113783 750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:35.113896 750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:35.115010 750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "5f2374b5371d43959e74035d64fb6c85"
format_stamp: "Formatted at 2026-08-07 08:42:35 on dist-test-slave-9zdj"
server_key: "6b982b9af5aa7306fe2c92b538fdbc6b"
server_key_iv: "9971bcbc458551adb56aa0c6224bf5f5"
server_key_version: "encryptionkey@0"
I20260807 08:42:35.115332 750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "5f2374b5371d43959e74035d64fb6c85"
format_stamp: "Formatted at 2026-08-07 08:42:35 on dist-test-slave-9zdj"
server_key: "6b982b9af5aa7306fe2c92b538fdbc6b"
server_key_iv: "9971bcbc458551adb56aa0c6224bf5f5"
server_key_version: "encryptionkey@0"
I20260807 08:42:35.116987 750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 08:42:35.118078 765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:35.118253 750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:35.118481 750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "5f2374b5371d43959e74035d64fb6c85"
format_stamp: "Formatted at 2026-08-07 08:42:35 on dist-test-slave-9zdj"
server_key: "6b982b9af5aa7306fe2c92b538fdbc6b"
server_key_iv: "9971bcbc458551adb56aa0c6224bf5f5"
server_key_version: "encryptionkey@0"
I20260807 08:42:35.118567 750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:35.134187 750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:35.137059 750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:35.137202 750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:35.137501 750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:35.137925 750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:35.137962 750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:35.137985 750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:35.137998 750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:35.145099 750 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:41139
I20260807 08:42:35.145321 886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:41139 every 8 connection(s)
I20260807 08:42:35.145635 750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:35.151782 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 750
I20260807 08:42:35.151958 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:42:35.152145 29106 external_mini_cluster.cc:1428] Setting key 41b201b0df80592cd406b89f12d79641
Aug 07 08:42:35 dist-test-slave-9zdj krb5kdc[486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092155, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:35.157838 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:35.146655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:38645 (local address 127.28.108.190:44307)
0807 08:42:35.146726 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:35.146728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:35.147016 (+ 288us) server_negotiation.cc:407] Connection header received
0807 08:42:35.148143 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:35.148145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:35.148157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:35.148220 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:35.148983 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.149578 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.150686 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.150880 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.150937 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:35.153882 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:35.153884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:35.153884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:35.153894 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:35.155664 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:35.156399 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:35.156400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:35.156401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:35.156468 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:35.156886 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:35.156887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:35.156887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:35.157011 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:35.157070 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:35.157488 (+ 418us) server_negotiation.cc:299] Negotiation successful
0807 08:42:35.157592 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 08:42:35.157964 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:35.146641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:44307 (local address 127.28.108.130:38645)
0807 08:42:35.146862 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:35.146903 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:35.148005 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:35.148389 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:35.148393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:35.148689 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:35.148900 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:35.148904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.149730 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.149732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:35.150608 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:35.150610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.150968 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.150969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:35.151086 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:35.151807 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:35.151816 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:35.153780 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:35.155862 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:35.155873 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:35.155876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:35.156251 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:35.156555 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:35.156556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:35.156557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:35.156807 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:35.157083 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:35.157090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:35.157287 (+ 197us) client_negotiation.cc:769] Sending connection context
0807 08:42:35.157414 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 08:42:35.157648 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":106,"threads_started":1}
I20260807 08:42:35.158538 888 heartbeater.cc:347] Connected to a master server at 127.28.108.190:44307
I20260807 08:42:35.158663 888 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:35.158921 888 heartbeater.cc:511] Master 127.28.108.190:44307 requested a full tablet report, sending...
I20260807 08:42:35.159441 539 ts_manager.cc:194] Registered new tserver with Master: 5f2374b5371d43959e74035d64fb6c85 (127.28.108.130:41139)
I20260807 08:42:35.160068 539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:38645
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:42:35.192049 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:44307
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:35.287736 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:35.288053 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:35.288105 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:35.289438 894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:35.289537 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:35.289584 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:35.291440 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:44307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:35.291796 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:35.292079 894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:35.298866 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:35.298887 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:35.299067 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:35.299170 894 server_base.cc:1034] running on GCE node
I20260807 08:42:35.299585 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:35.299968 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:35.301167 894 hybrid_clock.cc:648] HybridClock initialized: now 1786092155301140 us; error 46 us; skew 500 ppm
Aug 07 08:42:35 dist-test-slave-9zdj krb5kdc[486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092155, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:35.304461 894 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:42:35.305087 894 webserver.cc:460] Webserver started at http://127.28.108.131:32925/ using document root <none> and password file <none>
I20260807 08:42:35.305394 894 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:35.305476 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:35.305716 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:35.307087 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "7da54cef63e848ddae0bf91314f6ccaa"
format_stamp: "Formatted at 2026-08-07 08:42:35 on dist-test-slave-9zdj"
server_key: "627fdba6088ce0660cfc8e92da36490d"
server_key_iv: "5e6dbf48b3cc8be49d0167244542cab5"
server_key_version: "encryptionkey@0"
I20260807 08:42:35.307543 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "7da54cef63e848ddae0bf91314f6ccaa"
format_stamp: "Formatted at 2026-08-07 08:42:35 on dist-test-slave-9zdj"
server_key: "627fdba6088ce0660cfc8e92da36490d"
server_key_iv: "5e6dbf48b3cc8be49d0167244542cab5"
server_key_version: "encryptionkey@0"
I20260807 08:42:35.309263 894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:42:35.310586 910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:35.310921 894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:35.311074 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "7da54cef63e848ddae0bf91314f6ccaa"
format_stamp: "Formatted at 2026-08-07 08:42:35 on dist-test-slave-9zdj"
server_key: "627fdba6088ce0660cfc8e92da36490d"
server_key_iv: "5e6dbf48b3cc8be49d0167244542cab5"
server_key_version: "encryptionkey@0"
I20260807 08:42:35.311168 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:35.346791 894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:35.349884 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:35.350098 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:35.350399 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:35.350811 894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:35.350883 894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:35.350934 894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:35.350963 894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:35.358395 894 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:43169
I20260807 08:42:35.358459 1036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:43169 every 8 connection(s)
I20260807 08:42:35.358877 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:35.359894 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 894
I20260807 08:42:35.360010 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:35.360219 29106 external_mini_cluster.cc:1428] Setting key 4855f18c22a6ca4c26d6a4b8f01c6327
Aug 07 08:42:35 dist-test-slave-9zdj krb5kdc[486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092155, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:35.371260 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:35.359912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:53707 (local address 127.28.108.190:44307)
0807 08:42:35.359976 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:35.359979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:35.360372 (+ 393us) server_negotiation.cc:407] Connection header received
0807 08:42:35.361873 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:35.361874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:35.361886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:35.361937 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:35.362794 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.363366 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.364505 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.364721 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.364800 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:35.367549 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:35.367551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:35.367552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:35.367562 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:35.369483 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:35.370125 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:35.370126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:35.370127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:35.370204 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:35.370508 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:35.370509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:35.370509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:35.370650 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:35.370735 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:35.371046 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 08:42:35.371150 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 08:42:35.371500 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:35.359826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:44307 (local address 127.28.108.131:53707)
0807 08:42:35.360154 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:35.360208 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:35.361775 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:35.362000 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:35.362003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:35.362410 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:35.362681 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:35.362687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.363469 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.363470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:35.364415 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:35.364417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.364838 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.364840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:35.364958 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:35.365775 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:35.365785 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:35.367420 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:35.369608 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:35.369661 (+ 53us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:35.369664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:35.370009 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:35.370287 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:35.370288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:35.370289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:35.370424 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:35.370725 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:35.370733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:35.370898 (+ 165us) client_negotiation.cc:769] Sending connection context
0807 08:42:35.371010 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 08:42:35.371205 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":121,"threads_started":1}
I20260807 08:42:35.372080 1037 heartbeater.cc:347] Connected to a master server at 127.28.108.190:44307
I20260807 08:42:35.372205 1037 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:35.372416 1037 heartbeater.cc:511] Master 127.28.108.190:44307 requested a full tablet report, sending...
I20260807 08:42:35.372861 538 ts_manager.cc:194] Registered new tserver with Master: 7da54cef63e848ddae0bf91314f6ccaa (127.28.108.131:43169)
I20260807 08:42:35.373306 538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:53707
I20260807 08:42:35.383772 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 08:42:35 dist-test-slave-9zdj krb5kdc[486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 08:42:35 dist-test-slave-9zdj krb5kdc[486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
I20260807 08:42:35.414162 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:35.403869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.28.108.129:33967)
0807 08:42:35.404127 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:35.404132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:35.404146 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:42:35.404178 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:35.404180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:35.404199 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:35.404307 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:35.405003 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.405959 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.406881 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.407381 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.407457 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:35.410508 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:35.410521 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:35.410521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:35.410530 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:35.412687 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:35.413107 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:35.413111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:35.413111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:35.413190 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:35.413457 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:35.413458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:35.413458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:35.413690 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:35.413819 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:35.413969 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 08:42:35.414047 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":132,"threads_started":1}
Aug 07 08:42:35 dist-test-slave-9zdj krb5kdc[486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:35.424093 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:35.415924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.28.108.190:44307)
0807 08:42:35.415994 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:35.415996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:35.416035 (+ 39us) server_negotiation.cc:407] Connection header received
0807 08:42:35.416135 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:35.416136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:35.416148 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:35.416198 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:35.416736 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.417273 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.418161 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:35.418481 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:35.418553 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:35.421184 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:35.421186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:35.421186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:35.421195 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:35.422889 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:35.423253 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:35.423254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:35.423255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:35.423319 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:35.423537 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:35.423537 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:35.423538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:35.423710 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:35.423796 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:35.423913 (+ 117us) server_negotiation.cc:299] Negotiation successful
0807 08:42:35.423986 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260807 08:42:35.424432 538 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37236
I20260807 08:42:35.424893 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 605
I20260807 08:42:35.433725 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 750
I20260807 08:42:35.440083 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 894
I20260807 08:42:35.448015 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 504
2026-08-07T08:42:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2209 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:35 dist-test-slave-9zdj krb5kdc[1049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:35 dist-test-slave-9zdj krb5kdc[1049](info): set up 2 sockets
Aug 07 08:42:35 dist-test-slave-9zdj krb5kdc[1049](info): commencing operation
krb5kdc: starting...
W20260807 08:42:36.588529 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:36 dist-test-slave-9zdj krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:36.754961 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34873
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:34873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:36.849436 1064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:36.849820 1064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:36.849875 1064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:36.851433 1064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:36.851543 1064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:36.851572 1064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:36.851596 1064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:36.851629 1064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:36.853708 1064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:34873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:36.854111 1064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:36.854420 1064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:36.860703 1070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:36.860731 1069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:36.860749 1072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:36.861099 1064 server_base.cc:1034] running on GCE node
I20260807 08:42:36.861394 1064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:36.861809 1064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:36.863049 1064 hybrid_clock.cc:648] HybridClock initialized: now 1786092156863026 us; error 39 us; skew 500 ppm
Aug 07 08:42:36 dist-test-slave-9zdj krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:36.873914 1064 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:36.874420 1064 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:36.874514 1064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:36.874704 1064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:36.876195 1064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "d6c56e926d044720995d6d592a437665"
format_stamp: "Formatted at 2026-08-07 08:42:36 on dist-test-slave-9zdj"
server_key: "d903a709d27175b25908b3528d14d577"
server_key_iv: "1f34e2026931157c5998f03c353e1511"
server_key_version: "encryptionkey@0"
I20260807 08:42:36.876648 1064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "d6c56e926d044720995d6d592a437665"
format_stamp: "Formatted at 2026-08-07 08:42:36 on dist-test-slave-9zdj"
server_key: "d903a709d27175b25908b3528d14d577"
server_key_iv: "1f34e2026931157c5998f03c353e1511"
server_key_version: "encryptionkey@0"
I20260807 08:42:36.878538 1064 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260807 08:42:36.879455 1078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:36.879724 1064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:36.879860 1064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "d6c56e926d044720995d6d592a437665"
format_stamp: "Formatted at 2026-08-07 08:42:36 on dist-test-slave-9zdj"
server_key: "d903a709d27175b25908b3528d14d577"
server_key_iv: "1f34e2026931157c5998f03c353e1511"
server_key_version: "encryptionkey@0"
I20260807 08:42:36.879964 1064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:36.904484 1064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:36.907858 1064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:36.908109 1064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:36.913422 1064 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:34873
I20260807 08:42:36.913450 1134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:34873 every 8 connection(s)
I20260807 08:42:36.913888 1064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:36.915129 1135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:36.917809 1135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665: Bootstrap starting.
I20260807 08:42:36.918543 1135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:36.918898 1135 log.cc:824] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:36.919945 1135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665: No bootstrap required, opened a new log
I20260807 08:42:36.921857 1135 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c56e926d044720995d6d592a437665" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34873 } }
I20260807 08:42:36.922021 1135 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:36.922045 1135 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c56e926d044720995d6d592a437665, State: Initialized, Role: FOLLOWER
I20260807 08:42:36.922207 1135 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c56e926d044720995d6d592a437665" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34873 } }
I20260807 08:42:36.922289 1135 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:36.922315 1135 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:36.922348 1135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:36.923170 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 1064
I20260807 08:42:36.923280 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:36.923527 29106 external_mini_cluster.cc:1428] Setting key f3298d23f85b5f9873229978a73eff5d
I20260807 08:42:36.923887 1135 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c56e926d044720995d6d592a437665" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34873 } }
I20260807 08:42:36.924057 1135 leader_election.cc:302] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c56e926d044720995d6d592a437665; no voters:
I20260807 08:42:36.924363 1135 leader_election.cc:288] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:36.924417 1139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:36.924654 1139 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [term 1 LEADER]: Becoming Leader. State: Replica: d6c56e926d044720995d6d592a437665, State: Running, Role: LEADER
I20260807 08:42:36.924829 1139 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c56e926d044720995d6d592a437665" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34873 } }
I20260807 08:42:36.924888 1135 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:36.925428 1141 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6c56e926d044720995d6d592a437665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c56e926d044720995d6d592a437665" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34873 } } }
I20260807 08:42:36.925520 1141 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:36.926115 1142 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c56e926d044720995d6d592a437665. Latest consensus state: current_term: 1 leader_uuid: "d6c56e926d044720995d6d592a437665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c56e926d044720995d6d592a437665" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34873 } } }
I20260807 08:42:36.926297 1142 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:36.926795 1146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:36.927734 1146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:36.930045 1146 catalog_manager.cc:1347] Generated new cluster ID: 56aa21ede3474330a6254c6ad5eda435
I20260807 08:42:36.930121 1146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:42:36 dist-test-slave-9zdj krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:36.939076 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:36.924292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.28.108.190:34873)
0807 08:42:36.924580 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:36.924593 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:36.924655 (+ 62us) server_negotiation.cc:407] Connection header received
0807 08:42:36.924869 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:36.924878 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:36.924945 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:36.926233 (+ 1288us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:36.926381 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:36.928654 (+ 2273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:36.929960 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:36.930670 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:36.930816 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:36.934283 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:36.934289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:36.934290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:36.934307 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:36.936851 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:36.937428 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:36.937433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:36.937434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:36.937596 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:36.937884 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:36.937885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:36.937885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:36.938319 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:36.938503 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:36.938634 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 08:42:36.938816 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":133,"threads_started":1}
I20260807 08:42:36.945711 1146 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:36.946929 1146 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:36.954872 1146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665: Generated new TSK 0
I20260807 08:42:36.955260 1146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:42:36.995460 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34873
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:37.097021 1162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:37.097314 1162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:37.097347 1162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:37.098883 1162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:37.098968 1162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:37.099004 1162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:37.101030 1162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:37.101331 1162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:37.101569 1162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:37.108443 1168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:37.108541 1170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:37.108461 1167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:37.108875 1162 server_base.cc:1034] running on GCE node
I20260807 08:42:37.109181 1162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:37.109545 1162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:37.110827 1162 hybrid_clock.cc:648] HybridClock initialized: now 1786092157110808 us; error 35 us; skew 500 ppm
Aug 07 08:42:37 dist-test-slave-9zdj krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:37.114519 1162 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:42:37.115113 1162 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:37.115221 1162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:37.115489 1162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:37.117050 1162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "073ca5e1409142168a7a40c3e713fec5"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "edc8980ec9f1d9d6db76df02723e5614"
server_key_iv: "af673f57d0ffd324e8bdcb154caf67b4"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.117558 1162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "073ca5e1409142168a7a40c3e713fec5"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "edc8980ec9f1d9d6db76df02723e5614"
server_key_iv: "af673f57d0ffd324e8bdcb154caf67b4"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.119552 1162 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:42:37.120877 1176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:37.121393 1162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:37.121649 1162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "073ca5e1409142168a7a40c3e713fec5"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "edc8980ec9f1d9d6db76df02723e5614"
server_key_iv: "af673f57d0ffd324e8bdcb154caf67b4"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.121765 1162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:37.140188 1162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:37.143009 1162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:37.143211 1162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:37.143505 1162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:37.143895 1162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:37.143961 1162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:37.144001 1162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:37.144049 1162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:37.151849 1162 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:38035
I20260807 08:42:37.151939 1302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:38035 every 8 connection(s)
I20260807 08:42:37.152374 1162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:42:37.156423 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 1162
I20260807 08:42:37.156606 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:37.156824 29106 external_mini_cluster.cc:1428] Setting key c7e2b224e3dbf3fcf15cf52858147c3e
Aug 07 08:42:37 dist-test-slave-9zdj krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:37.167668 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.153455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:34827 (local address 127.28.108.190:34873)
0807 08:42:37.153514 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:37.153516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:37.153774 (+ 258us) server_negotiation.cc:407] Connection header received
0807 08:42:37.154500 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:37.154502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:37.154521 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:37.154601 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:37.156092 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.156850 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.158418 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.158821 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.158922 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.162575 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:37.162578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:37.162578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:37.162590 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:37.165364 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:37.166224 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:37.166225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:37.166226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:37.166402 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:37.166804 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:37.166805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:37.166805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:37.166952 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:37.167018 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:37.167480 (+ 462us) server_negotiation.cc:299] Negotiation successful
0807 08:42:37.167591 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 08:42:37.168198 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.153363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34873 (local address 127.28.108.129:34827)
0807 08:42:37.153573 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:37.153645 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:37.154234 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:37.154819 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:37.154823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:37.155160 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:37.155897 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.155905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.157021 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.157023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.158283 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.158287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.158976 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.158977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.159196 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.160058 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:37.160073 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:37.162364 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:37.165540 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:37.165557 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:37.165561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:37.166090 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:37.166497 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:37.166499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:37.166499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:37.166710 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:37.167030 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:37.167039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:37.167287 (+ 248us) client_negotiation.cc:769] Sending connection context
0807 08:42:37.167432 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 08:42:37.167661 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
I20260807 08:42:37.168869 1303 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34873
I20260807 08:42:37.169006 1303 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:37.169260 1303 heartbeater.cc:511] Master 127.28.108.190:34873 requested a full tablet report, sending...
I20260807 08:42:37.169998 1098 ts_manager.cc:194] Registered new tserver with Master: 073ca5e1409142168a7a40c3e713fec5 (127.28.108.129:38035)
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
I20260807 08:42:37.170750 1098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:34827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:42:37.199054 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34873
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:37.293131 1308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:37.293396 1308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:37.293426 1308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:37.294811 1308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:37.294880 1308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:37.294912 1308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:37.296614 1308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:37.296921 1308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:37.297215 1308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:37.303324 1313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:37.303224 1314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:37.303464 1308 server_base.cc:1034] running on GCE node
W20260807 08:42:37.303224 1316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:37.303933 1308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:37.304275 1308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:37.305550 1308 hybrid_clock.cc:648] HybridClock initialized: now 1786092157305559 us; error 66 us; skew 500 ppm
Aug 07 08:42:37 dist-test-slave-9zdj krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:37.309191 1308 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:42:37.309859 1308 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:37.309954 1308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:37.310137 1308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:37.311304 1308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "c210557d01094c158f963d5c89ead679"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "e887cb85ccae4db538d36777d6810211"
server_key_iv: "ba2c12f144fba87151ec2229ab8ff27f"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.311720 1308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "c210557d01094c158f963d5c89ead679"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "e887cb85ccae4db538d36777d6810211"
server_key_iv: "ba2c12f144fba87151ec2229ab8ff27f"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.313247 1308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:42:37.314564 1322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:37.314849 1308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:37.314987 1308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "c210557d01094c158f963d5c89ead679"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "e887cb85ccae4db538d36777d6810211"
server_key_iv: "ba2c12f144fba87151ec2229ab8ff27f"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.315182 1308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:37.328665 1308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:37.331558 1308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:37.331768 1308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:37.332065 1308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:37.332451 1308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:37.332511 1308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:37.332547 1308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:37.332576 1308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:37.339265 1308 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:46565
I20260807 08:42:37.339305 1442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:46565 every 8 connection(s)
I20260807 08:42:37.339630 1308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:37.346952 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 1308
I20260807 08:42:37.347061 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
Aug 07 08:42:37 dist-test-slave-9zdj krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:37.347311 29106 external_mini_cluster.cc:1428] Setting key c2ade1afe684679f12f94d5dfcab283b
I20260807 08:42:37.351651 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.340478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:52559 (local address 127.28.108.190:34873)
0807 08:42:37.340554 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:37.340556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:37.340818 (+ 262us) server_negotiation.cc:407] Connection header received
0807 08:42:37.341349 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:37.341351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:37.341363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:37.341422 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:37.342549 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.343304 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.344660 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.344919 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.344996 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.347976 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:37.347977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:37.347978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:37.347985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:37.349871 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:37.350493 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:37.350496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:37.350497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:37.350566 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:37.350884 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:37.350885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:37.350885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:37.351019 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:37.351097 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:37.351432 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 08:42:37.351533 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 08:42:37.351862 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.340470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34873 (local address 127.28.108.130:52559)
0807 08:42:37.340679 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:37.340722 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:37.341228 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:37.341481 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:37.341485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:37.341791 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:37.342318 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.342323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.343457 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.343458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.344515 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.344518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.345040 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.345041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.345277 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.345944 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:37.345954 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:37.347875 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:37.349969 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:37.349977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:37.349980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:37.350383 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:37.350654 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:37.350655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:37.350655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:37.350806 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:37.351098 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:37.351107 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:37.351283 (+ 176us) client_negotiation.cc:769] Sending connection context
0807 08:42:37.351398 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 08:42:37.351586 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":99,"threads_started":1}
I20260807 08:42:37.352433 1443 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34873
I20260807 08:42:37.352566 1443 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:37.352733 1443 heartbeater.cc:511] Master 127.28.108.190:34873 requested a full tablet report, sending...
I20260807 08:42:37.353224 1098 ts_manager.cc:194] Registered new tserver with Master: c210557d01094c158f963d5c89ead679 (127.28.108.130:46565)
I20260807 08:42:37.353847 1098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:52559
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:42:37.386893 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34873
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:37.479128 1449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:37.479375 1449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:37.479404 1449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:37.480720 1449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:37.480789 1449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:37.480821 1449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:37.482771 1449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:37.483085 1449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:37.483354 1449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:37.489360 1454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:37.489574 1455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:37.489632 1449 server_base.cc:1034] running on GCE node
W20260807 08:42:37.489809 1457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:37.490106 1449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:37.490515 1449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:37.491787 1449 hybrid_clock.cc:648] HybridClock initialized: now 1786092157491751 us; error 65 us; skew 500 ppm
Aug 07 08:42:37 dist-test-slave-9zdj krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:37.499323 1449 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:42:37.499861 1449 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:37.499955 1449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:37.500145 1449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:37.501524 1449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "38ab6053022f4f95ad53bb2d398166f1"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "27a75748ca8462d791fac5704f03ebc4"
server_key_iv: "6000c3f96d745d6bf28c24835d602abf"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.502053 1449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "38ab6053022f4f95ad53bb2d398166f1"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "27a75748ca8462d791fac5704f03ebc4"
server_key_iv: "6000c3f96d745d6bf28c24835d602abf"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.503806 1449 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:42:37.504971 1463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:37.505357 1449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:37.505523 1449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "38ab6053022f4f95ad53bb2d398166f1"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "27a75748ca8462d791fac5704f03ebc4"
server_key_iv: "6000c3f96d745d6bf28c24835d602abf"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.505681 1449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:37.521173 1449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:37.524839 1449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:37.525106 1449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:37.525571 1449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:37.526140 1449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:37.526199 1449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:37.526244 1449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:37.526284 1449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:37.534227 1449 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:37047
I20260807 08:42:37.534451 1585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:37047 every 8 connection(s)
I20260807 08:42:37.534690 1449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
Aug 07 08:42:37 dist-test-slave-9zdj krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:37.544857 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 1449
I20260807 08:42:37.545006 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:37.545415 29106 external_mini_cluster.cc:1428] Setting key 0d8d7d62e0ae48fdbbd0ef5a6529c1ee
I20260807 08:42:37.546413 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.535779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:51217 (local address 127.28.108.190:34873)
0807 08:42:37.535837 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:37.535839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:37.536069 (+ 230us) server_negotiation.cc:407] Connection header received
0807 08:42:37.536602 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:37.536603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:37.536615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:37.536686 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:37.537813 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.538341 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.539613 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.539808 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.539869 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.542771 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:37.542773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:37.542773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:37.542781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:37.544303 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:37.544863 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:37.544864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:37.544865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:37.544948 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:37.545340 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:37.545342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:37.545342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:37.545644 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:37.545731 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:37.546183 (+ 452us) server_negotiation.cc:299] Negotiation successful
0807 08:42:37.546308 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 08:42:37.546697 1588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.535644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34873 (local address 127.28.108.131:51217)
0807 08:42:37.535866 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:37.535916 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:37.536480 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:37.536757 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:37.536762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:37.537073 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:37.537682 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.537689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.538534 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.538535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.539522 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.539526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.539933 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.539934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.540070 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.540811 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:37.540821 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:37.542635 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:37.544412 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:37.544422 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:37.544425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:37.544749 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:37.545069 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:37.545070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:37.545071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:37.545235 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:37.545734 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:37.545743 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:37.545956 (+ 213us) client_negotiation.cc:769] Sending connection context
0807 08:42:37.546107 (+ 151us) client_negotiation.cc:240] Negotiation successful
0807 08:42:37.546403 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":127,"threads_started":1}
I20260807 08:42:37.547192 1586 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34873
I20260807 08:42:37.547324 1586 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:37.547503 1586 heartbeater.cc:511] Master 127.28.108.190:34873 requested a full tablet report, sending...
I20260807 08:42:37.547988 1098 ts_manager.cc:194] Registered new tserver with Master: 38ab6053022f4f95ad53bb2d398166f1 (127.28.108.131:37047)
I20260807 08:42:37.548403 1098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:51217
I20260807 08:42:37.558017 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:42:37.568735 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.559700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.28.108.190:34873)
0807 08:42:37.559817 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:37.559819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:37.559974 (+ 155us) server_negotiation.cc:407] Connection header received
0807 08:42:37.560005 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:37.560006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:37.560018 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:37.560090 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:37.560941 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.561507 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.562878 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.563138 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.563216 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.565150 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:37.565151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:37.565152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:37.565170 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:37.567101 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:37.567602 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:37.567602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:37.567603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:37.567669 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:37.568011 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:37.568012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:37.568012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:37.568276 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:37.568385 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:37.568572 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 08:42:37.568660 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 08:42:37.571007 1098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:37.572927 1098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:37.580770 1598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.576001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:37047 (local address 127.0.0.1:34308)
0807 08:42:37.576246 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:37.576256 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:37.576303 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:37.576698 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:37.576699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:37.576701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:37.576781 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.576784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.578192 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.578193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.579514 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.579517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.580424 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.580424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.580515 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.580523 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 08:42:37.580568 (+ 45us) client_negotiation.cc:240] Negotiation successful
0807 08:42:37.580659 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":105,"thread_start_us":85,"threads_started":1}
I20260807 08:42:37.580796 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.576040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.28.108.129:38035)
0807 08:42:37.576254 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:37.576259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:37.576272 (+ 13us) server_negotiation.cc:407] Connection header received
0807 08:42:37.576336 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:37.576338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:37.576357 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:37.576446 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:37.576942 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.577814 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.579700 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.580265 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.580370 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.580441 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:37.580615 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 08:42:37.580661 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":105,"threads_started":1}
I20260807 08:42:37.580787 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.575724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:46565 (local address 127.0.0.1:58238)
0807 08:42:37.575885 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:37.575935 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:37.576003 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:37.576728 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:37.576729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:37.576731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:37.576785 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.576787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.578247 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.578248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.579371 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.579374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.580421 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.580422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.580515 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.580525 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 08:42:37.580557 (+ 32us) client_negotiation.cc:240] Negotiation successful
0807 08:42:37.580658 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":53,"thread_start_us":61,"threads_started":1}
I20260807 08:42:37.581043 1599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.575932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.28.108.130:46565)
0807 08:42:37.576338 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:37.576343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:37.576554 (+ 211us) server_negotiation.cc:407] Connection header received
0807 08:42:37.576601 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:37.576604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:37.576622 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:37.576728 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:37.576861 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.578076 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.579495 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.580303 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.580717 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.580785 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:37.580813 (+ 28us) server_negotiation.cc:299] Negotiation successful
0807 08:42:37.580871 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":102,"threads_started":1}
I20260807 08:42:37.581120 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.575772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:38035 (local address 127.0.0.1:54902)
0807 08:42:37.576138 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:37.576151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:37.576208 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:37.576594 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:37.576598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:37.576601 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:37.576879 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.576881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.578044 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.578046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.579620 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.579622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.580370 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.580371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.580467 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.580483 (+ 16us) client_negotiation.cc:769] Sending connection context
0807 08:42:37.580593 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 08:42:37.580659 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":38,"thread_start_us":118,"threads_started":1}
I20260807 08:42:37.581941 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.576236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.28.108.131:37047)
0807 08:42:37.576503 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:37.576507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:37.576517 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:37.576562 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:37.576565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:37.576583 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:37.576874 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:37.576963 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.578087 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.579622 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.580310 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.580596 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.581767 (+ 1171us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:37.581802 (+ 35us) server_negotiation.cc:299] Negotiation successful
0807 08:42:37.581854 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":164,"threads_started":1}
I20260807 08:42:37.582154 1230 tablet_service.cc:1467] Processing CreateTablet for tablet 6b08f351e1cd4f41a782c513e74202aa (DEFAULT_TABLE table=test-table [id=a660746b64c14338b34f3f2b2bf4197f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:37.582297 1374 tablet_service.cc:1467] Processing CreateTablet for tablet 6b08f351e1cd4f41a782c513e74202aa (DEFAULT_TABLE table=test-table [id=a660746b64c14338b34f3f2b2bf4197f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:37.582625 1230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b08f351e1cd4f41a782c513e74202aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:37.582697 1374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b08f351e1cd4f41a782c513e74202aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:37.582744 1515 tablet_service.cc:1467] Processing CreateTablet for tablet 6b08f351e1cd4f41a782c513e74202aa (DEFAULT_TABLE table=test-table [id=a660746b64c14338b34f3f2b2bf4197f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:37.583070 1515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b08f351e1cd4f41a782c513e74202aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:37.586855 1604 tablet_bootstrap.cc:492] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5: Bootstrap starting.
I20260807 08:42:37.587343 1605 tablet_bootstrap.cc:492] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1: Bootstrap starting.
I20260807 08:42:37.587343 1606 tablet_bootstrap.cc:492] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679: Bootstrap starting.
I20260807 08:42:37.588160 1606 tablet_bootstrap.cc:654] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:37.588166 1605 tablet_bootstrap.cc:654] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:37.588187 1604 tablet_bootstrap.cc:654] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:37.588887 1604 log.cc:824] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:37.588977 1606 log.cc:824] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:37.588977 1605 log.cc:824] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:37.590070 1604 tablet_bootstrap.cc:492] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5: No bootstrap required, opened a new log
I20260807 08:42:37.590163 1604 ts_tablet_manager.cc:1397] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:42:37.590175 1606 tablet_bootstrap.cc:492] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679: No bootstrap required, opened a new log
I20260807 08:42:37.590351 1606 ts_tablet_manager.cc:1397] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:42:37.592522 1604 raft_consensus.cc:348] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073ca5e1409142168a7a40c3e713fec5" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38035 } } peers { permanent_uuid: "38ab6053022f4f95ad53bb2d398166f1" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37047 } } peers { permanent_uuid: "c210557d01094c158f963d5c89ead679" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46565 } }
I20260807 08:42:37.592522 1606 raft_consensus.cc:348] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073ca5e1409142168a7a40c3e713fec5" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38035 } } peers { permanent_uuid: "38ab6053022f4f95ad53bb2d398166f1" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37047 } } peers { permanent_uuid: "c210557d01094c158f963d5c89ead679" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46565 } }
I20260807 08:42:37.592767 1606 raft_consensus.cc:374] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:37.592767 1604 raft_consensus.cc:374] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:37.592810 1606 raft_consensus.cc:729] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c210557d01094c158f963d5c89ead679, State: Initialized, Role: FOLLOWER
I20260807 08:42:37.592810 1604 raft_consensus.cc:729] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 073ca5e1409142168a7a40c3e713fec5, State: Initialized, Role: FOLLOWER
I20260807 08:42:37.592978 1605 tablet_bootstrap.cc:492] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1: No bootstrap required, opened a new log
I20260807 08:42:37.592991 1606 consensus_queue.cc:260] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073ca5e1409142168a7a40c3e713fec5" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38035 } } peers { permanent_uuid: "38ab6053022f4f95ad53bb2d398166f1" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37047 } } peers { permanent_uuid: "c210557d01094c158f963d5c89ead679" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46565 } }
I20260807 08:42:37.592991 1604 consensus_queue.cc:260] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073ca5e1409142168a7a40c3e713fec5" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38035 } } peers { permanent_uuid: "38ab6053022f4f95ad53bb2d398166f1" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37047 } } peers { permanent_uuid: "c210557d01094c158f963d5c89ead679" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46565 } }
I20260807 08:42:37.593129 1605 ts_tablet_manager.cc:1397] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260807 08:42:37.593348 1606 ts_tablet_manager.cc:1428] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:42:37.593384 1303 heartbeater.cc:503] Master 127.28.108.190:34873 was elected leader, sending a full tablet report...
I20260807 08:42:37.593705 1443 heartbeater.cc:503] Master 127.28.108.190:34873 was elected leader, sending a full tablet report...
I20260807 08:42:37.593950 1604 ts_tablet_manager.cc:1428] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260807 08:42:37.595595 1605 raft_consensus.cc:348] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073ca5e1409142168a7a40c3e713fec5" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38035 } } peers { permanent_uuid: "38ab6053022f4f95ad53bb2d398166f1" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37047 } } peers { permanent_uuid: "c210557d01094c158f963d5c89ead679" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46565 } }
I20260807 08:42:37.595782 1605 raft_consensus.cc:374] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:37.595808 1605 raft_consensus.cc:729] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ab6053022f4f95ad53bb2d398166f1, State: Initialized, Role: FOLLOWER
I20260807 08:42:37.595970 1605 consensus_queue.cc:260] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073ca5e1409142168a7a40c3e713fec5" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38035 } } peers { permanent_uuid: "38ab6053022f4f95ad53bb2d398166f1" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37047 } } peers { permanent_uuid: "c210557d01094c158f963d5c89ead679" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46565 } }
I20260807 08:42:37.596359 1586 heartbeater.cc:503] Master 127.28.108.190:34873 was elected leader, sending a full tablet report...
I20260807 08:42:37.596356 1605 ts_tablet_manager.cc:1428] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20260807 08:42:37.653335 1304 tablet.cc:2369] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:37.741909 1610 raft_consensus.cc:482] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:37.742231 1610 raft_consensus.cc:504] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073ca5e1409142168a7a40c3e713fec5" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38035 } } peers { permanent_uuid: "38ab6053022f4f95ad53bb2d398166f1" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37047 } } peers { permanent_uuid: "c210557d01094c158f963d5c89ead679" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46565 } }
I20260807 08:42:37.742784 1610 leader_election.cc:288] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38ab6053022f4f95ad53bb2d398166f1 (127.28.108.131:37047), c210557d01094c158f963d5c89ead679 (127.28.108.130:46565)
I20260807 08:42:37.746935 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.743165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:58047 (local address 127.28.108.131:37047)
0807 08:42:37.743258 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:37.743261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:37.743347 (+ 86us) server_negotiation.cc:407] Connection header received
0807 08:42:37.743414 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:37.743415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:37.743426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:37.743493 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:37.743750 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.744390 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.745896 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.746412 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.746481 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.746490 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:37.746793 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 08:42:37.746851 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 08:42:37.746944 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.743013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:37047 (local address 127.28.108.129:58047)
0807 08:42:37.743247 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:37.743261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:37.743340 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:37.743549 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:37.743550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:37.743554 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:37.743663 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.743667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.744562 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.744563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.745794 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.745798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.746521 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.746522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.746614 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.746619 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:37.746690 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 08:42:37.746734 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260807 08:42:37.746990 1599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.743165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:44489 (local address 127.28.108.130:46565)
0807 08:42:37.743270 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:37.743272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:37.743519 (+ 247us) server_negotiation.cc:407] Connection header received
0807 08:42:37.743554 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:37.743556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:37.743568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:37.743651 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:37.743908 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.744607 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.746030 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.746524 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.746615 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.746625 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:37.746856 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 08:42:37.746910 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 08:42:37.746958 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.743013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:46565 (local address 127.28.108.129:44489)
0807 08:42:37.743256 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:37.743270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:37.743398 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:37.743662 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:37.743663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:37.743664 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:37.743798 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.743799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.744737 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.744738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.745920 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:37.745924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.746632 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.746633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:37.746742 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.746748 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:42:37.746821 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 08:42:37.746868 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":23,"thread_start_us":65,"threads_started":1}
I20260807 08:42:37.747473 1396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b08f351e1cd4f41a782c513e74202aa" candidate_uuid: "073ca5e1409142168a7a40c3e713fec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c210557d01094c158f963d5c89ead679" is_pre_election: true
I20260807 08:42:37.747715 1396 raft_consensus.cc:2454] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 073ca5e1409142168a7a40c3e713fec5 in term 0.
I20260807 08:42:37.747700 1535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b08f351e1cd4f41a782c513e74202aa" candidate_uuid: "073ca5e1409142168a7a40c3e713fec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ab6053022f4f95ad53bb2d398166f1" is_pre_election: true
I20260807 08:42:37.747869 1535 raft_consensus.cc:2454] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 073ca5e1409142168a7a40c3e713fec5 in term 0.
I20260807 08:42:37.748059 1181 leader_election.cc:302] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 073ca5e1409142168a7a40c3e713fec5, c210557d01094c158f963d5c89ead679; no voters:
I20260807 08:42:37.748402 1610 raft_consensus.cc:2781] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:37.748497 1610 raft_consensus.cc:482] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:37.748531 1610 raft_consensus.cc:3037] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:37.749347 1610 raft_consensus.cc:504] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073ca5e1409142168a7a40c3e713fec5" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38035 } } peers { permanent_uuid: "38ab6053022f4f95ad53bb2d398166f1" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37047 } } peers { permanent_uuid: "c210557d01094c158f963d5c89ead679" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46565 } }
I20260807 08:42:37.749529 1610 leader_election.cc:288] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [CANDIDATE]: Term 1 election: Requested vote from peers 38ab6053022f4f95ad53bb2d398166f1 (127.28.108.131:37047), c210557d01094c158f963d5c89ead679 (127.28.108.130:46565)
I20260807 08:42:37.749792 1535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b08f351e1cd4f41a782c513e74202aa" candidate_uuid: "073ca5e1409142168a7a40c3e713fec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ab6053022f4f95ad53bb2d398166f1"
I20260807 08:42:37.749879 1535 raft_consensus.cc:3037] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:37.750593 1535 raft_consensus.cc:2454] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 073ca5e1409142168a7a40c3e713fec5 in term 1.
I20260807 08:42:37.750749 1396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b08f351e1cd4f41a782c513e74202aa" candidate_uuid: "073ca5e1409142168a7a40c3e713fec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c210557d01094c158f963d5c89ead679"
I20260807 08:42:37.750856 1396 raft_consensus.cc:3037] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:37.750820 1177 leader_election.cc:302] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 073ca5e1409142168a7a40c3e713fec5, 38ab6053022f4f95ad53bb2d398166f1; no voters:
I20260807 08:42:37.751024 1610 raft_consensus.cc:2781] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:37.751220 1610 raft_consensus.cc:686] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [term 1 LEADER]: Becoming Leader. State: Replica: 073ca5e1409142168a7a40c3e713fec5, State: Running, Role: LEADER
I20260807 08:42:37.751438 1610 consensus_queue.cc:237] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073ca5e1409142168a7a40c3e713fec5" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38035 } } peers { permanent_uuid: "38ab6053022f4f95ad53bb2d398166f1" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37047 } } peers { permanent_uuid: "c210557d01094c158f963d5c89ead679" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46565 } }
I20260807 08:42:37.751864 1396 raft_consensus.cc:2454] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 073ca5e1409142168a7a40c3e713fec5 in term 1.
I20260807 08:42:37.752444 1098 catalog_manager.cc:5495] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 073ca5e1409142168a7a40c3e713fec5 (127.28.108.129). New cstate: current_term: 1 leader_uuid: "073ca5e1409142168a7a40c3e713fec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073ca5e1409142168a7a40c3e713fec5" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38ab6053022f4f95ad53bb2d398166f1" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c210557d01094c158f963d5c89ead679" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46565 } health_report { overall_health: UNKNOWN } } }
I20260807 08:42:37.765604 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.762574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.28.108.129:38035)
0807 08:42:37.762654 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:37.762656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:37.762703 (+ 47us) server_negotiation.cc:407] Connection header received
0807 08:42:37.762750 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:37.762751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:37.762773 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:37.762839 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:37.763050 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.763707 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.764654 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:37.764854 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:37.764920 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:37.765120 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:37.765340 (+ 220us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:37.765430 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:37.765512 (+ 82us) server_negotiation.cc:299] Negotiation successful
0807 08:42:37.765536 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 08:42:37.768138 1396 raft_consensus.cc:1261] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679 [term 1 FOLLOWER]: Refusing update from remote peer 073ca5e1409142168a7a40c3e713fec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:37.768138 1535 raft_consensus.cc:1261] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1 [term 1 FOLLOWER]: Refusing update from remote peer 073ca5e1409142168a7a40c3e713fec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:37.768654 1615 consensus_queue.cc:1035] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c210557d01094c158f963d5c89ead679" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:37.768887 1610 consensus_queue.cc:1035] T 6b08f351e1cd4f41a782c513e74202aa P 073ca5e1409142168a7a40c3e713fec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ab6053022f4f95ad53bb2d398166f1" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:37.772552 1620 mvcc.cc:205] Tried to move back new op lower bound from 7315833478214557696 to 7315833478150770688. Current Snapshot: MvccSnapshot[applied={T|T < 7315833478214557696}]
I20260807 08:42:37.772830 1619 mvcc.cc:205] Tried to move back new op lower bound from 7315833478214557696 to 7315833478150770688. Current Snapshot: MvccSnapshot[applied={T|T < 7315833478214557696}]
I20260807 08:42:37.775110 1098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:37.775225 1098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:37.776109 1098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6c56e926d044720995d6d592a437665: Sending DeleteTablet for 3 replicas of tablet 6b08f351e1cd4f41a782c513e74202aa
I20260807 08:42:37.776515 1374 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b08f351e1cd4f41a782c513e74202aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:37 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:58238
I20260807 08:42:37.776579 1230 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b08f351e1cd4f41a782c513e74202aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:37 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:54902
I20260807 08:42:37.776644 1515 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b08f351e1cd4f41a782c513e74202aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:37 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:34308
I20260807 08:42:37.776958 1632 tablet_replica.cc:330] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1: stopping tablet replica
I20260807 08:42:37.777127 1632 raft_consensus.cc:2229] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:37.777221 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 1162
I20260807 08:42:37.777271 1632 raft_consensus.cc:2258] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:37.783317 1632 ts_tablet_manager.cc:1905] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:37.785532 1629 tablet_replica.cc:330] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679: stopping tablet replica
I20260807 08:42:37.785547 1632 ts_tablet_manager.cc:1918] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:37.785679 1632 log.cc:1196] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/6b08f351e1cd4f41a782c513e74202aa
I20260807 08:42:37.785809 1629 raft_consensus.cc:2229] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:37.785989 1629 raft_consensus.cc:2258] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:37.786028 1632 ts_tablet_manager.cc:1939] T 6b08f351e1cd4f41a782c513e74202aa P 38ab6053022f4f95ad53bb2d398166f1: Deleting consensus metadata
I20260807 08:42:37.786535 1629 ts_tablet_manager.cc:1905] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:37.786815 1079 catalog_manager.cc:4841] TS 38ab6053022f4f95ad53bb2d398166f1 (127.28.108.131:37047): tablet 6b08f351e1cd4f41a782c513e74202aa (table test-table [id=a660746b64c14338b34f3f2b2bf4197f]) successfully deleted
I20260807 08:42:37.788199 1629 ts_tablet_manager.cc:1918] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:37.788275 1629 log.cc:1196] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/6b08f351e1cd4f41a782c513e74202aa
I20260807 08:42:37.788617 1629 ts_tablet_manager.cc:1939] T 6b08f351e1cd4f41a782c513e74202aa P c210557d01094c158f963d5c89ead679: Deleting consensus metadata
W20260807 08:42:37.788791 1079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:38035 (error 108)
I20260807 08:42:37.789072 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 1308
I20260807 08:42:37.791142 1081 catalog_manager.cc:4841] TS c210557d01094c158f963d5c89ead679 (127.28.108.130:46565): tablet 6b08f351e1cd4f41a782c513e74202aa (table test-table [id=a660746b64c14338b34f3f2b2bf4197f]) successfully deleted
I20260807 08:42:37.792217 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.792023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:38035 (local address 127.0.0.1:54922)
0807 08:42:37.792117 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:37.792174 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260807 08:42:37.792394 1079 catalog_manager.cc:4584] TS 073ca5e1409142168a7a40c3e713fec5 (127.28.108.129:38035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b08f351e1cd4f41a782c513e74202aa: Network error: Client connection negotiation failed: client connection to 127.28.108.129:38035: connect: Connection refused (error 111)
I20260807 08:42:37.797353 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 1449
I20260807 08:42:37.804708 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 1064
2026-08-07T08:42:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2356 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T08:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:37Z Disabled control of system clock
I20260807 08:42:37.840229 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:35501
--webserver_interface=127.28.108.190
--webserver_port=0
--builtin_ntp_servers=127.28.108.148:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:35501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 08:42:37.927820 1637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:37.928122 1637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:37.928164 1637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:37.929533 1637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:37.929603 1637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:37.929623 1637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:37.929672 1637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:37.929701 1637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:37.931311 1637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:35501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:35501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:37.931659 1637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:37.932016 1637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:37.937906 1643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:37.937870 1642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:37.937875 1645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:37.938228 1637 server_base.cc:1034] running on GCE node
I20260807 08:42:37.938730 1637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:37.939112 1637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:37.940392 1637 hybrid_clock.cc:648] HybridClock initialized: now 1786092157940340 us; error 62 us; skew 500 ppm
I20260807 08:42:37.942054 1637 webserver.cc:460] Webserver started at http://127.28.108.190:34327/ using document root <none> and password file <none>
I20260807 08:42:37.942365 1637 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:37.942458 1637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:37.942642 1637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:37.944059 1637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "1f41cf038f71486192c2a610d650dc20"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "9c75bd262ec9e678a51004041be8d42d"
server_key_iv: "57578467aa7ef47be83dd0788cd10e5d"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.944499 1637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "1f41cf038f71486192c2a610d650dc20"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "9c75bd262ec9e678a51004041be8d42d"
server_key_iv: "57578467aa7ef47be83dd0788cd10e5d"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.946347 1637 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 08:42:37.947409 1651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:37.947646 1637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:37.947784 1637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "1f41cf038f71486192c2a610d650dc20"
format_stamp: "Formatted at 2026-08-07 08:42:37 on dist-test-slave-9zdj"
server_key: "9c75bd262ec9e678a51004041be8d42d"
server_key_iv: "57578467aa7ef47be83dd0788cd10e5d"
server_key_version: "encryptionkey@0"
I20260807 08:42:37.947881 1637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:37.952230 1637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:37.952572 1637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:37.952749 1637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:37.958308 1637 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:35501
I20260807 08:42:37.958371 1708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:35501 every 8 connection(s)
I20260807 08:42:37.958811 1637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:37.961865 1709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:37.964572 1709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20: Bootstrap starting.
I20260807 08:42:37.965298 1709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:37.965734 1709 log.cc:824] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:37.966603 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 1637
I20260807 08:42:37.966728 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:37.966907 29106 external_mini_cluster.cc:1428] Setting key b65f970c04e3cc528f3a2e2e31c2fe07
I20260807 08:42:37.966907 1709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20: No bootstrap required, opened a new log
I20260807 08:42:37.969594 1709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f41cf038f71486192c2a610d650dc20" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35501 } }
I20260807 08:42:37.969895 1709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:37.969947 1709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f41cf038f71486192c2a610d650dc20, State: Initialized, Role: FOLLOWER
I20260807 08:42:37.970111 1709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f41cf038f71486192c2a610d650dc20" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35501 } }
I20260807 08:42:37.970278 1709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:37.970325 1709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:37.970376 1709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:37.970623 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:37.967601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.28.108.190:35501)
0807 08:42:37.967871 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:37.967881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:37.967920 (+ 39us) server_negotiation.cc:407] Connection header received
0807 08:42:37.968573 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:37.968584 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:37.968646 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:37.968822 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:37.969771 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:37.969776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:37.969776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:37.969781 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:37.969837 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:37.969851 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:37.969941 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:37.970315 (+ 374us) server_negotiation.cc:299] Negotiation successful
0807 08:42:37.970409 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":123,"threads_started":1}
I20260807 08:42:37.971740 1709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f41cf038f71486192c2a610d650dc20" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35501 } }
I20260807 08:42:37.971887 1709 leader_election.cc:302] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f41cf038f71486192c2a610d650dc20; no voters:
I20260807 08:42:37.972114 1709 leader_election.cc:288] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:37.972301 1714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:37.972441 1709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:37.972551 1714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [term 1 LEADER]: Becoming Leader. State: Replica: 1f41cf038f71486192c2a610d650dc20, State: Running, Role: LEADER
I20260807 08:42:37.972744 1714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f41cf038f71486192c2a610d650dc20" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35501 } }
I20260807 08:42:37.973193 1715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f41cf038f71486192c2a610d650dc20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f41cf038f71486192c2a610d650dc20" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35501 } } }
I20260807 08:42:37.973309 1715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:37.973752 1716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f41cf038f71486192c2a610d650dc20. Latest consensus state: current_term: 1 leader_uuid: "1f41cf038f71486192c2a610d650dc20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f41cf038f71486192c2a610d650dc20" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35501 } } }
I20260807 08:42:37.973847 1716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20 [sys.catalog]: This master's current role is: LEADER
W20260807 08:42:37.974572 1731 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:42:37.974646 1731 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:42:37.974709 1723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:37.974867 1723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:37.976526 1723 catalog_manager.cc:1347] Generated new cluster ID: f1c260aba63544d9afd02c6ce1a741f0
I20260807 08:42:37.976598 1723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:37.985977 1723 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:37.986234 1723 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:37.991899 1723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20: Generated new TSK 0
I20260807 08:42:37.992168 1723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:42:37.997088 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35501
--builtin_ntp_servers=127.28.108.148:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 08:42:38.089986 1735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:38.090281 1735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:38.090325 1735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:38.091732 1735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:38.091825 1735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:38.091873 1735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:38.093678 1735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:38.094027 1735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:38.094393 1735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:38.100275 1740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:38.100275 1743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:38.100345 1741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:38.100751 1735 server_base.cc:1034] running on GCE node
I20260807 08:42:38.101007 1735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:38.101300 1735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:38.102519 1735 hybrid_clock.cc:648] HybridClock initialized: now 1786092158102468 us; error 72 us; skew 500 ppm
I20260807 08:42:38.104467 1735 webserver.cc:460] Webserver started at http://127.28.108.129:43355/ using document root <none> and password file <none>
I20260807 08:42:38.104863 1735 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:38.104959 1735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:38.105180 1735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:38.106626 1735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "ee8eb9fb92a64a18920aea36da978a42"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "f70ec0a7f2317f5b799d56573877b071"
server_key_iv: "2c483a4eb56f4cf1bea0d2a49e7f35b3"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.107100 1735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "ee8eb9fb92a64a18920aea36da978a42"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "f70ec0a7f2317f5b799d56573877b071"
server_key_iv: "2c483a4eb56f4cf1bea0d2a49e7f35b3"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.108785 1735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 08:42:38.109881 1749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:38.110110 1735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:38.110247 1735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "ee8eb9fb92a64a18920aea36da978a42"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "f70ec0a7f2317f5b799d56573877b071"
server_key_iv: "2c483a4eb56f4cf1bea0d2a49e7f35b3"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.110348 1735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:38.114548 1735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:38.114931 1735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:38.115097 1735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:38.115339 1735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:38.115738 1735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:38.115778 1735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:38.115803 1735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:38.115857 1735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:38.123417 1735 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:44909
I20260807 08:42:38.123476 1871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:44909 every 8 connection(s)
I20260807 08:42:38.123946 1735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:42:38.127800 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.124828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:34283 (local address 127.28.108.190:35501)
0807 08:42:38.124894 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.124896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.125182 (+ 286us) server_negotiation.cc:407] Connection header received
0807 08:42:38.125842 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.125844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.125851 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.125909 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:38.127395 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:38.127396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:38.127396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:38.127402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:38.127418 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:38.127423 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:38.127492 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:38.127712 (+ 220us) server_negotiation.cc:299] Negotiation successful
0807 08:42:38.127730 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 08:42:38.127926 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.124822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35501 (local address 127.28.108.129:34283)
0807 08:42:38.125028 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.125066 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:38.125716 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:38.125980 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:38.125984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:38.125993 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:38.127163 (+ 1170us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:42:38.127163 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:38.127196 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:42:38.127232 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:38.127536 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:38.127544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:38.127547 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:38.127629 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 08:42:38.127691 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260807 08:42:38.128662 1873 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35501
I20260807 08:42:38.128793 1873 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:38.129016 1873 heartbeater.cc:511] Master 127.28.108.190:35501 requested a full tablet report, sending...
I20260807 08:42:38.129736 1669 ts_manager.cc:194] Registered new tserver with Master: ee8eb9fb92a64a18920aea36da978a42 (127.28.108.129:44909)
I20260807 08:42:38.134021 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 1735
I20260807 08:42:38.134132 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:38.134322 29106 external_mini_cluster.cc:1428] Setting key dd24ea8dd81b557153b77c7d125d9a5b
I20260807 08:42:38.135506 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35501
--builtin_ntp_servers=127.28.108.148:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 08:42:38.234807 1876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:38.235085 1876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:38.235113 1876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:38.236893 1876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:38.236989 1876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:38.237049 1876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:38.239667 1876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:38.240048 1876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:38.240490 1876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:38.247509 1881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:38.247673 1884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:38.248050 1876 server_base.cc:1034] running on GCE node
W20260807 08:42:38.247501 1882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:38.248498 1876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:38.248893 1876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:38.250224 1876 hybrid_clock.cc:648] HybridClock initialized: now 1786092158250229 us; error 61 us; skew 500 ppm
I20260807 08:42:38.252445 1876 webserver.cc:460] Webserver started at http://127.28.108.130:39647/ using document root <none> and password file <none>
I20260807 08:42:38.252815 1876 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:38.252910 1876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:38.253101 1876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:38.254601 1876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "3ecd84400f53434aa76a9f2c5f6096b2"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "5bbc67263aed20823b320d60a28981ec"
server_key_iv: "c0f7bec767439d2cb00d209584c4ee28"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.254984 1876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "3ecd84400f53434aa76a9f2c5f6096b2"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "5bbc67263aed20823b320d60a28981ec"
server_key_iv: "c0f7bec767439d2cb00d209584c4ee28"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.256855 1876 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 08:42:38.258160 1891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:38.258574 1876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:38.258673 1876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "3ecd84400f53434aa76a9f2c5f6096b2"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "5bbc67263aed20823b320d60a28981ec"
server_key_iv: "c0f7bec767439d2cb00d209584c4ee28"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.258752 1876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:38.262876 1876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:38.263172 1876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:38.263276 1876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:38.263463 1876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:38.263824 1876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:38.263860 1876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:38.263883 1876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:38.263896 1876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:38.270756 1876 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:35063
I20260807 08:42:38.270810 2014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:35063 every 8 connection(s)
I20260807 08:42:38.271178 1876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:38.272840 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 1876
I20260807 08:42:38.273006 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:42:38.273218 29106 external_mini_cluster.cc:1428] Setting key 71964d0c10c70aa81118274a88a3abc6
I20260807 08:42:38.274473 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35501
--builtin_ntp_servers=127.28.108.148:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 08:42:38.275069 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.272021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:39135 (local address 127.28.108.190:35501)
0807 08:42:38.272069 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.272071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.272423 (+ 352us) server_negotiation.cc:407] Connection header received
0807 08:42:38.273233 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.273235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.273242 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.273309 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:38.274807 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:38.274809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:38.274809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:38.274814 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:38.274831 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:38.274836 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:38.274893 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:38.274982 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 08:42:38.274999 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 08:42:38.275285 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.272061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35501 (local address 127.28.108.130:39135)
0807 08:42:38.272267 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.272311 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:38.273107 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:38.273389 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:38.273394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:38.273405 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:38.274587 (+ 1182us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:42:38.274588 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:38.274667 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:42:38.274710 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:38.274896 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:38.274904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:38.274908 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:38.274994 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 08:42:38.275068 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260807 08:42:38.276602 2015 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35501
I20260807 08:42:38.276741 2015 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:38.276916 2015 heartbeater.cc:511] Master 127.28.108.190:35501 requested a full tablet report, sending...
I20260807 08:42:38.277379 1669 ts_manager.cc:194] Registered new tserver with Master: 3ecd84400f53434aa76a9f2c5f6096b2 (127.28.108.130:35063)
W20260807 08:42:38.364513 2018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:38.364806 2018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:38.364852 2018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:38.366350 2018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:38.366460 2018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:38.366509 2018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:38.368060 2018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:38.368423 2018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:38.368791 2018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:38.374818 2024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:38.374886 2018 server_base.cc:1034] running on GCE node
W20260807 08:42:38.374845 2026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:38.374817 2023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:38.375376 2018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:38.375712 2018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:38.376938 2018 hybrid_clock.cc:648] HybridClock initialized: now 1786092158376918 us; error 34 us; skew 500 ppm
I20260807 08:42:38.378772 2018 webserver.cc:460] Webserver started at http://127.28.108.131:39913/ using document root <none> and password file <none>
I20260807 08:42:38.379083 2018 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:38.379143 2018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:38.379304 2018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:38.380599 2018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "3ba40680163b43aaa9c1db8b98673066"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "eab3aad3b67cb17952fd0007d97b1d15"
server_key_iv: "33ba2578ddd572d44cdb30cc9fc5b916"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.381058 2018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "3ba40680163b43aaa9c1db8b98673066"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "eab3aad3b67cb17952fd0007d97b1d15"
server_key_iv: "33ba2578ddd572d44cdb30cc9fc5b916"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.382864 2018 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260807 08:42:38.384013 2032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:38.384258 2018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:38.384473 2018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "3ba40680163b43aaa9c1db8b98673066"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "eab3aad3b67cb17952fd0007d97b1d15"
server_key_iv: "33ba2578ddd572d44cdb30cc9fc5b916"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.384580 2018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:38.388782 2018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:38.389132 2018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:38.389305 2018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:38.389578 2018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:38.390007 2018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:38.390071 2018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:38.390121 2018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:38.390157 2018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:38.397082 2018 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:40811
I20260807 08:42:38.397133 2157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:40811 every 8 connection(s)
I20260807 08:42:38.397555 2018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:38.401278 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.398533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:51289 (local address 127.28.108.190:35501)
0807 08:42:38.398615 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.398617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.398915 (+ 298us) server_negotiation.cc:407] Connection header received
0807 08:42:38.399462 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.399463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.399472 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.399533 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:38.400970 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:38.400972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:38.400973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:38.400978 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:38.400996 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:38.401001 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:38.401058 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:38.401175 (+ 117us) server_negotiation.cc:299] Negotiation successful
0807 08:42:38.401196 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 08:42:38.401665 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 2018
I20260807 08:42:38.401804 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:38.402022 29106 external_mini_cluster.cc:1428] Setting key c09980f99c569b5378d72a2df351373f
I20260807 08:42:38.402381 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.398531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35501 (local address 127.28.108.131:51289)
0807 08:42:38.398738 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.398776 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:38.399324 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:38.399620 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:38.399623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:38.399632 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:38.400781 (+ 1149us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:42:38.400782 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:38.400818 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:42:38.400852 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:38.401065 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:38.401072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:38.401075 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:38.401173 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 08:42:38.401251 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260807 08:42:38.403136 2158 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35501
I20260807 08:42:38.403244 2158 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:38.403410 2158 heartbeater.cc:511] Master 127.28.108.190:35501 requested a full tablet report, sending...
I20260807 08:42:38.403837 1669 ts_manager.cc:194] Registered new tserver with Master: 3ba40680163b43aaa9c1db8b98673066 (127.28.108.131:40811)
I20260807 08:42:38.414522 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:42:38.417549 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.416123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.28.108.190:35501)
0807 08:42:38.416210 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.416213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.416287 (+ 74us) server_negotiation.cc:407] Connection header received
0807 08:42:38.416367 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.416368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.416375 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.416424 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:38.417210 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:38.417212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:38.417212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:38.417217 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:38.417237 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:38.417242 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:38.417332 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:38.417412 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 08:42:38.417425 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 08:42:38.419430 1669 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:38.421240 1669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:38.426729 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.424525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:40811 (local address 127.0.0.1:40610)
0807 08:42:38.424884 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.424952 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:38.425021 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:38.425515 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:38.425524 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:38.425532 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:38.426181 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:42:38.426182 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:38.426201 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:42:38.426239 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:38.426496 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:38.426503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:38.426507 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:38.426593 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 08:42:38.426617 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":93,"threads_started":1}
I20260807 08:42:38.427114 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.424532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:35063 (local address 127.0.0.1:39826)
0807 08:42:38.425153 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.425165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:38.425211 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:38.425963 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:38.425964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:38.425966 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:38.426581 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:42:38.426582 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:38.426590 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:42:38.426603 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:38.426968 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:38.426968 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:38.426969 (+ 1us) client_negotiation.cc:769] Sending connection context
0807 08:42:38.427013 (+ 44us) client_negotiation.cc:240] Negotiation successful
0807 08:42:38.427024 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":138,"threads_started":1}
I20260807 08:42:38.427114 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.424631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.28.108.131:40811)
0807 08:42:38.425340 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.425344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.425353 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:42:38.425404 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.425407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.425418 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.425511 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:38.426336 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:38.426346 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:38.426347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:38.426352 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:38.426391 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:38.426420 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:38.426506 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:38.426999 (+ 493us) server_negotiation.cc:299] Negotiation successful
0807 08:42:38.427019 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":176,"threads_started":1}
I20260807 08:42:38.427181 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.425054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.28.108.129:44909)
0807 08:42:38.425303 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.425309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.425323 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:42:38.425425 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.425430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.425440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.425513 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:38.426727 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:38.426731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:38.426731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:38.426734 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:38.426788 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:38.426807 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:38.426866 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:38.427066 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 08:42:38.427084 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":124,"threads_started":1}
I20260807 08:42:38.427707 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.424625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.28.108.130:35063)
0807 08:42:38.425070 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.425076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.425275 (+ 199us) server_negotiation.cc:407] Connection header received
0807 08:42:38.425310 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.425314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.425326 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.425410 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:38.426677 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:38.426692 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:38.426693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:38.426697 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:38.426738 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:38.426768 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:38.426940 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:38.427574 (+ 634us) server_negotiation.cc:299] Negotiation successful
0807 08:42:38.427593 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":128,"threads_started":1}
I20260807 08:42:38.427887 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.424791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:44909 (local address 127.0.0.1:50290)
0807 08:42:38.425155 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.425166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:38.425209 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:38.425973 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:38.425975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:38.425978 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:38.426509 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:42:38.426510 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:38.426518 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:42:38.426533 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:38.426898 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:38.426900 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:38.426900 (+ 0us) client_negotiation.cc:769] Sending connection context
0807 08:42:38.426948 (+ 48us) client_negotiation.cc:240] Negotiation successful
0807 08:42:38.426965 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":91,"threads_started":1}
I20260807 08:42:38.427968 2084 tablet_service.cc:1467] Processing CreateTablet for tablet b7335f7416ac499e982cb08451de1e7a (DEFAULT_TABLE table=test-table [id=4fc5d8ca3f51430db2fd61614f69be94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:38.428423 2084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7335f7416ac499e982cb08451de1e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:38.428663 1803 tablet_service.cc:1467] Processing CreateTablet for tablet b7335f7416ac499e982cb08451de1e7a (DEFAULT_TABLE table=test-table [id=4fc5d8ca3f51430db2fd61614f69be94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:38.428663 1945 tablet_service.cc:1467] Processing CreateTablet for tablet b7335f7416ac499e982cb08451de1e7a (DEFAULT_TABLE table=test-table [id=4fc5d8ca3f51430db2fd61614f69be94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:38.429036 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7335f7416ac499e982cb08451de1e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:38.429036 1803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7335f7416ac499e982cb08451de1e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:38.431365 2173 tablet_bootstrap.cc:492] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066: Bootstrap starting.
I20260807 08:42:38.432066 2173 tablet_bootstrap.cc:654] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:38.432382 2173 log.cc:824] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:38.433182 2175 tablet_bootstrap.cc:492] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42: Bootstrap starting.
I20260807 08:42:38.433292 2173 tablet_bootstrap.cc:492] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066: No bootstrap required, opened a new log
I20260807 08:42:38.433365 2173 ts_tablet_manager.cc:1397] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 08:42:38.433820 2176 tablet_bootstrap.cc:492] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2: Bootstrap starting.
I20260807 08:42:38.434036 2175 tablet_bootstrap.cc:654] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:38.434441 2175 log.cc:824] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:38.434615 2176 tablet_bootstrap.cc:654] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:38.435003 2176 log.cc:824] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:38.435432 2175 tablet_bootstrap.cc:492] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42: No bootstrap required, opened a new log
I20260807 08:42:38.435508 2175 ts_tablet_manager.cc:1397] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260807 08:42:38.435948 2176 tablet_bootstrap.cc:492] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2: No bootstrap required, opened a new log
I20260807 08:42:38.436002 2176 ts_tablet_manager.cc:1397] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:42:38.435961 2173 raft_consensus.cc:348] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 35063 } } peers { permanent_uuid: "3ba40680163b43aaa9c1db8b98673066" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40811 } } peers { permanent_uuid: "ee8eb9fb92a64a18920aea36da978a42" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44909 } }
I20260807 08:42:38.436132 2173 raft_consensus.cc:374] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:38.436167 2173 raft_consensus.cc:729] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba40680163b43aaa9c1db8b98673066, State: Initialized, Role: FOLLOWER
I20260807 08:42:38.436296 2173 consensus_queue.cc:260] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 35063 } } peers { permanent_uuid: "3ba40680163b43aaa9c1db8b98673066" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40811 } } peers { permanent_uuid: "ee8eb9fb92a64a18920aea36da978a42" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44909 } }
I20260807 08:42:38.436645 2158 heartbeater.cc:503] Master 127.28.108.190:35501 was elected leader, sending a full tablet report...
I20260807 08:42:38.436580 2173 ts_tablet_manager.cc:1428] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260807 08:42:38.438550 2175 raft_consensus.cc:348] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 35063 } } peers { permanent_uuid: "3ba40680163b43aaa9c1db8b98673066" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40811 } } peers { permanent_uuid: "ee8eb9fb92a64a18920aea36da978a42" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44909 } }
I20260807 08:42:38.438717 2175 raft_consensus.cc:374] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:38.438766 2175 raft_consensus.cc:729] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8eb9fb92a64a18920aea36da978a42, State: Initialized, Role: FOLLOWER
I20260807 08:42:38.438886 2175 consensus_queue.cc:260] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 35063 } } peers { permanent_uuid: "3ba40680163b43aaa9c1db8b98673066" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40811 } } peers { permanent_uuid: "ee8eb9fb92a64a18920aea36da978a42" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44909 } }
I20260807 08:42:38.439169 2175 ts_tablet_manager.cc:1428] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260807 08:42:38.439271 1873 heartbeater.cc:503] Master 127.28.108.190:35501 was elected leader, sending a full tablet report...
I20260807 08:42:38.439491 2176 raft_consensus.cc:348] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 35063 } } peers { permanent_uuid: "3ba40680163b43aaa9c1db8b98673066" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40811 } } peers { permanent_uuid: "ee8eb9fb92a64a18920aea36da978a42" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44909 } }
I20260807 08:42:38.439692 2176 raft_consensus.cc:374] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:38.439740 2176 raft_consensus.cc:729] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ecd84400f53434aa76a9f2c5f6096b2, State: Initialized, Role: FOLLOWER
I20260807 08:42:38.439886 2176 consensus_queue.cc:260] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 35063 } } peers { permanent_uuid: "3ba40680163b43aaa9c1db8b98673066" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40811 } } peers { permanent_uuid: "ee8eb9fb92a64a18920aea36da978a42" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44909 } }
I20260807 08:42:38.440193 2176 ts_tablet_manager.cc:1428] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 08:42:38.440263 2015 heartbeater.cc:503] Master 127.28.108.190:35501 was elected leader, sending a full tablet report...
W20260807 08:42:38.521858 2016 tablet.cc:2369] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:38.528630 2182 raft_consensus.cc:482] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:38.528863 2182 raft_consensus.cc:504] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 35063 } } peers { permanent_uuid: "3ba40680163b43aaa9c1db8b98673066" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40811 } } peers { permanent_uuid: "ee8eb9fb92a64a18920aea36da978a42" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44909 } }
I20260807 08:42:38.529402 2182 leader_election.cc:288] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ba40680163b43aaa9c1db8b98673066 (127.28.108.131:40811), ee8eb9fb92a64a18920aea36da978a42 (127.28.108.129:44909)
I20260807 08:42:38.531666 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.529779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:40811 (local address 127.28.108.130:56007)
0807 08:42:38.530007 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.530024 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:38.530092 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:38.530590 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:38.530591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:38.530594 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:38.531151 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:42:38.531152 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:38.531161 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:42:38.531179 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:38.531484 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:38.531486 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:38.531486 (+ 0us) client_negotiation.cc:769] Sending connection context
0807 08:42:38.531557 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 08:42:38.531573 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260807 08:42:38.531894 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.530114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:56007 (local address 127.28.108.131:40811)
0807 08:42:38.530395 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.530398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.530408 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:38.530443 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.530444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.530450 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.530507 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:38.531284 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:38.531285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:38.531286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:38.531292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:38.531345 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:38.531352 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:38.531476 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:38.531778 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 08:42:38.531796 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
I20260807 08:42:38.532202 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.530342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:36639 (local address 127.28.108.129:44909)
0807 08:42:38.530434 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.530437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.530447 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:38.530505 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.530506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.530512 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.530613 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:38.531745 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:38.531747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:38.531747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:38.531752 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:38.531802 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:38.531809 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:38.531848 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:38.532084 (+ 236us) server_negotiation.cc:299] Negotiation successful
0807 08:42:38.532103 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 08:42:38.532565 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.529915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:44909 (local address 127.28.108.130:36639)
0807 08:42:38.530195 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.530207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:38.530262 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:38.530723 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:38.530724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:38.530727 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:38.531577 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:42:38.531578 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:38.531610 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:42:38.531623 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:38.531929 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:38.531930 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:38.531931 (+ 1us) client_negotiation.cc:769] Sending connection context
0807 08:42:38.532042 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 08:42:38.532427 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
I20260807 08:42:38.532960 2107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7335f7416ac499e982cb08451de1e7a" candidate_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba40680163b43aaa9c1db8b98673066" is_pre_election: true
I20260807 08:42:38.532972 1824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7335f7416ac499e982cb08451de1e7a" candidate_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee8eb9fb92a64a18920aea36da978a42" is_pre_election: true
I20260807 08:42:38.533133 2107 raft_consensus.cc:2454] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ecd84400f53434aa76a9f2c5f6096b2 in term 0.
I20260807 08:42:38.533156 1824 raft_consensus.cc:2454] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ecd84400f53434aa76a9f2c5f6096b2 in term 0.
I20260807 08:42:38.533454 1892 leader_election.cc:302] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba40680163b43aaa9c1db8b98673066, 3ecd84400f53434aa76a9f2c5f6096b2; no voters:
I20260807 08:42:38.533679 2182 raft_consensus.cc:2781] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:38.533767 2182 raft_consensus.cc:482] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:38.533798 2182 raft_consensus.cc:3037] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:38.534852 2182 raft_consensus.cc:504] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 35063 } } peers { permanent_uuid: "3ba40680163b43aaa9c1db8b98673066" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40811 } } peers { permanent_uuid: "ee8eb9fb92a64a18920aea36da978a42" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44909 } }
I20260807 08:42:38.535022 2182 leader_election.cc:288] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [CANDIDATE]: Term 1 election: Requested vote from peers 3ba40680163b43aaa9c1db8b98673066 (127.28.108.131:40811), ee8eb9fb92a64a18920aea36da978a42 (127.28.108.129:44909)
I20260807 08:42:38.540685 2107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7335f7416ac499e982cb08451de1e7a" candidate_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba40680163b43aaa9c1db8b98673066"
I20260807 08:42:38.540832 2107 raft_consensus.cc:3037] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:38.540824 1824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7335f7416ac499e982cb08451de1e7a" candidate_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee8eb9fb92a64a18920aea36da978a42"
I20260807 08:42:38.540970 1824 raft_consensus.cc:3037] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:38.542039 2107 raft_consensus.cc:2454] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ecd84400f53434aa76a9f2c5f6096b2 in term 1.
I20260807 08:42:38.542330 1824 raft_consensus.cc:2454] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ecd84400f53434aa76a9f2c5f6096b2 in term 1.
I20260807 08:42:38.542819 1892 leader_election.cc:302] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ecd84400f53434aa76a9f2c5f6096b2, ee8eb9fb92a64a18920aea36da978a42; no voters:
I20260807 08:42:38.543022 2182 raft_consensus.cc:2781] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:38.543274 2182 raft_consensus.cc:686] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 1 LEADER]: Becoming Leader. State: Replica: 3ecd84400f53434aa76a9f2c5f6096b2, State: Running, Role: LEADER
I20260807 08:42:38.543417 2182 consensus_queue.cc:237] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 35063 } } peers { permanent_uuid: "3ba40680163b43aaa9c1db8b98673066" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40811 } } peers { permanent_uuid: "ee8eb9fb92a64a18920aea36da978a42" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44909 } }
I20260807 08:42:38.544504 1669 catalog_manager.cc:5495] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ecd84400f53434aa76a9f2c5f6096b2 (127.28.108.130). New cstate: current_term: 1 leader_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecd84400f53434aa76a9f2c5f6096b2" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 35063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ba40680163b43aaa9c1db8b98673066" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8eb9fb92a64a18920aea36da978a42" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44909 } health_report { overall_health: UNKNOWN } } }
I20260807 08:42:38.577364 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.575900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.28.108.130:35063)
0807 08:42:38.576005 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.576007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.576018 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:42:38.576074 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.576075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.576091 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.576154 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:38.576873 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:38.576875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:38.576875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:38.576880 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:38.576897 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:38.576902 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:38.576957 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:38.577102 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 08:42:38.577125 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 08:42:38.579821 2107 raft_consensus.cc:1261] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066 [term 1 FOLLOWER]: Refusing update from remote peer 3ecd84400f53434aa76a9f2c5f6096b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:38.579857 1824 raft_consensus.cc:1261] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42 [term 1 FOLLOWER]: Refusing update from remote peer 3ecd84400f53434aa76a9f2c5f6096b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:38.580222 2184 consensus_queue.cc:1035] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8eb9fb92a64a18920aea36da978a42" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 44909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:38.580381 2182 consensus_queue.cc:1035] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba40680163b43aaa9c1db8b98673066" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:38.583889 2187 mvcc.cc:205] Tried to move back new op lower bound from 7315833481539563520 to 7315833481395400704. Current Snapshot: MvccSnapshot[applied={T|T < 7315833481539563520}]
I20260807 08:42:38.584045 2189 mvcc.cc:205] Tried to move back new op lower bound from 7315833481539563520 to 7315833481395400704. Current Snapshot: MvccSnapshot[applied={T|T < 7315833481539563520}]
I20260807 08:42:38.584398 2188 mvcc.cc:205] Tried to move back new op lower bound from 7315833481539563520 to 7315833481395400704. Current Snapshot: MvccSnapshot[applied={T|T < 7315833481539563520}]
I20260807 08:42:38.587297 1669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:38.587431 1669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:38.588279 1669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f41cf038f71486192c2a610d650dc20: Sending DeleteTablet for 3 replicas of tablet b7335f7416ac499e982cb08451de1e7a
I20260807 08:42:38.588670 1945 tablet_service.cc:1514] Processing DeleteTablet for tablet b7335f7416ac499e982cb08451de1e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:38 UTC) from {username='slave'} at 127.0.0.1:39826
I20260807 08:42:38.588719 2084 tablet_service.cc:1514] Processing DeleteTablet for tablet b7335f7416ac499e982cb08451de1e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:38 UTC) from {username='slave'} at 127.0.0.1:40610
I20260807 08:42:38.588749 1803 tablet_service.cc:1514] Processing DeleteTablet for tablet b7335f7416ac499e982cb08451de1e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:38 UTC) from {username='slave'} at 127.0.0.1:50290
I20260807 08:42:38.589003 2198 tablet_replica.cc:330] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2: stopping tablet replica
I20260807 08:42:38.589197 2199 tablet_replica.cc:330] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42: stopping tablet replica
I20260807 08:42:38.589221 2200 tablet_replica.cc:330] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066: stopping tablet replica
I20260807 08:42:38.589222 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 1735
I20260807 08:42:38.589344 2199 raft_consensus.cc:2229] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:38.589465 2199 raft_consensus.cc:2258] T b7335f7416ac499e982cb08451de1e7a P ee8eb9fb92a64a18920aea36da978a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:38.589597 2198 raft_consensus.cc:2229] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:38.589854 2198 raft_consensus.cc:2258] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:38.591056 2198 ts_tablet_manager.cc:1905] T b7335f7416ac499e982cb08451de1e7a P 3ecd84400f53434aa76a9f2c5f6096b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:38.591379 2200 raft_consensus.cc:2229] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:38.591596 2200 raft_consensus.cc:2258] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:38.592538 2200 ts_tablet_manager.cc:1905] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 08:42:38.599288 1652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:44909 (error 108)
I20260807 08:42:38.599939 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.599769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:44909 (local address 127.0.0.1:50294)
0807 08:42:38.599841 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.599892 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 08:42:38.600008 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 1876
W20260807 08:42:38.600220 1652 catalog_manager.cc:4584] TS ee8eb9fb92a64a18920aea36da978a42 (127.28.108.129:44909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7335f7416ac499e982cb08451de1e7a: Network error: Client connection negotiation failed: client connection to 127.28.108.129:44909: connect: Connection refused (error 111)
I20260807 08:42:38.637636 2200 ts_tablet_manager.cc:1918] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:38.637801 2200 log.cc:1196] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/b7335f7416ac499e982cb08451de1e7a
I20260807 08:42:38.638172 2200 ts_tablet_manager.cc:1939] T b7335f7416ac499e982cb08451de1e7a P 3ba40680163b43aaa9c1db8b98673066: Deleting consensus metadata
I20260807 08:42:38.640843 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 2018
I20260807 08:42:38.641036 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.640853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:35063 (local address 127.0.0.1:39846)
0807 08:42:38.640935 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.640985 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.130:35063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 08:42:38.652935 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.652733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:44909 (local address 127.0.0.1:50304)
0807 08:42:38.652829 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.652889 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 08:42:38.653780 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.653487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:44909 (local address 127.0.0.1:50320)
0807 08:42:38.653555 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.653590 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 08:42:38.658751 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 1637
I20260807 08:42:38.658949 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.658771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:40811 (local address 127.0.0.1:40624)
0807 08:42:38.658835 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.658900 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.131:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
2026-08-07T08:42:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (853 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 08:42:38.683830 29106 mini_oidc.cc:152] Starting JWKS server
I20260807 08:42:38.685191 29106 webserver.cc:460] Webserver started at https://127.0.0.1:36653/ using document root <none> and password file <none>
I20260807 08:42:38.685325 29106 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 08:42:38.685531 29106 webserver.cc:460] Webserver started at http://127.0.0.1:42601/ using document root <none> and password file <none>
2026-08-07T08:42:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:38Z Disabled control of system clock
I20260807 08:42:38.696095 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34935
--webserver_interface=127.28.108.190
--webserver_port=0
--builtin_ntp_servers=127.28.108.148:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:34935
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36653/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/testchainca.cert with env {}
W20260807 08:42:38.790091 2208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:38.790364 2208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:38.790393 2208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:38.791870 2208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:38.791932 2208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:38.791945 2208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:38.791958 2208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:38.791971 2208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 08:42:38.791985 2208 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 08:42:38.791996 2208 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36653/jwks.json
I20260807 08:42:38.793963 2208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:34935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36653/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:38.794248 2208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:38.794445 2208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:42:38.800251 2208 server_base.cc:1034] running on GCE node
W20260807 08:42:38.800364 2214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:38.800252 2217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:38.800415 2215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:38.800916 2208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:38.801314 2208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:38.802497 2208 hybrid_clock.cc:648] HybridClock initialized: now 1786092158802454 us; error 56 us; skew 500 ppm
I20260807 08:42:38.804219 2208 webserver.cc:460] Webserver started at http://127.28.108.190:42937/ using document root <none> and password file <none>
I20260807 08:42:38.804546 2208 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:38.804602 2208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:38.804800 2208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:38.806092 2208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "b00c604cf475446cad3b3867168b35e2"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "35b11f559716f531a3bb224a45911aa0"
server_key_iv: "f67fdc1cc6d2d49395ba25992d3caf06"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.806497 2208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "b00c604cf475446cad3b3867168b35e2"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "35b11f559716f531a3bb224a45911aa0"
server_key_iv: "f67fdc1cc6d2d49395ba25992d3caf06"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.808136 2208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:42:38.809579 2223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:38.809870 2208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:38.809986 2208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "b00c604cf475446cad3b3867168b35e2"
format_stamp: "Formatted at 2026-08-07 08:42:38 on dist-test-slave-9zdj"
server_key: "35b11f559716f531a3bb224a45911aa0"
server_key_iv: "f67fdc1cc6d2d49395ba25992d3caf06"
server_key_version: "encryptionkey@0"
I20260807 08:42:38.810091 2208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:38.840111 2208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:38.840540 2208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:38.840816 2208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:38.845664 2208 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:34935
I20260807 08:42:38.845705 2278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:34935 every 8 connection(s)
I20260807 08:42:38.846102 2208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:38.847215 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:38.849753 2280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2: Bootstrap starting.
I20260807 08:42:38.850363 2280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:38.850632 2280 log.cc:824] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:38.851429 2280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2: No bootstrap required, opened a new log
I20260807 08:42:38.853240 2280 raft_consensus.cc:348] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00c604cf475446cad3b3867168b35e2" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34935 } }
I20260807 08:42:38.853345 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 2208
I20260807 08:42:38.853535 2280 raft_consensus.cc:374] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:38.853614 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:38.853595 2280 raft_consensus.cc:729] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00c604cf475446cad3b3867168b35e2, State: Initialized, Role: FOLLOWER
I20260807 08:42:38.853811 2280 consensus_queue.cc:260] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00c604cf475446cad3b3867168b35e2" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34935 } }
I20260807 08:42:38.853909 2280 raft_consensus.cc:388] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:38.853957 2280 raft_consensus.cc:482] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:38.853973 29106 external_mini_cluster.cc:1428] Setting key 1f9b357fbd3cdf1b899108606fbb308a
I20260807 08:42:38.854007 2280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:38.861963 2284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.854591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.28.108.190:34935)
0807 08:42:38.854868 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.854879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.854916 (+ 37us) server_negotiation.cc:407] Connection header received
0807 08:42:38.855055 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.855062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.855143 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.855317 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:38.856143 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:38.857547 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:38.858794 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:38.859965 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:38.860095 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:38.861061 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:38.861067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:38.861068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:38.861278 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:38.861318 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:38.861331 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:38.861397 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:38.861506 (+ 109us) server_negotiation.cc:299] Negotiation successful
0807 08:42:38.861668 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
I20260807 08:42:38.863404 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:42:38.928660 2280 raft_consensus.cc:504] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00c604cf475446cad3b3867168b35e2" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34935 } }
I20260807 08:42:38.928960 2280 leader_election.cc:302] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b00c604cf475446cad3b3867168b35e2; no voters:
I20260807 08:42:38.929313 2280 leader_election.cc:288] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:38.929479 2285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:38.929726 2285 raft_consensus.cc:686] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [term 1 LEADER]: Becoming Leader. State: Replica: b00c604cf475446cad3b3867168b35e2, State: Running, Role: LEADER
I20260807 08:42:38.929823 2280 sys_catalog.cc:564] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:38.929867 2285 consensus_queue.cc:237] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00c604cf475446cad3b3867168b35e2" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34935 } }
I20260807 08:42:38.930898 2286 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b00c604cf475446cad3b3867168b35e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00c604cf475446cad3b3867168b35e2" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34935 } } }
I20260807 08:42:38.931020 2286 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:38.931334 2287 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b00c604cf475446cad3b3867168b35e2. Latest consensus state: current_term: 1 leader_uuid: "b00c604cf475446cad3b3867168b35e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00c604cf475446cad3b3867168b35e2" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34935 } } }
I20260807 08:42:38.931432 2287 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2 [sys.catalog]: This master's current role is: LEADER
W20260807 08:42:38.932372 2300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:42:38.932444 2300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:42:38.932504 2294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:38.932684 2294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:38.934535 2294 catalog_manager.cc:1347] Generated new cluster ID: fe3a2b65752745299fe5ae2d4731fb6b
I20260807 08:42:38.934629 2294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:38.956925 2294 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:38.957872 2294 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:38.962570 2294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b00c604cf475446cad3b3867168b35e2: Generated new TSK 0
I20260807 08:42:38.962831 2294 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 08:42:38.999706 2311 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 08:42:38.999917 2311 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 08:42:39.000403 2284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:38.982598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.28.108.190:34935)
0807 08:42:38.982656 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:38.982658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:38.982820 (+ 162us) server_negotiation.cc:407] Connection header received
0807 08:42:38.982925 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:38.982926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:38.982941 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:38.983042 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0807 08:42:38.983414 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:38.984210 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:38.985193 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:38.985513 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:38.985590 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:38.985978 (+ 388us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 08:42:38.999785 (+ 13807us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 08:42:39.000341 (+ 556us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260807 08:42:39.000512 2284 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 08:42:39.000597 2310 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:38.982483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34935 (local address 127.0.0.1:42646)
0807 08:42:38.982727 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:38.982743 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:38.982823 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:38.983101 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:38.983102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:38.983104 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0807 08:42:38.983313 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:38.983316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:38.984356 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:38.984357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:38.985064 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:38.985067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:38.985704 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:38.985706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:38.985798 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:38.985831 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 08:42:38.999969 (+ 14138us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 08:42:39.000117 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.108.190:34935: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":128,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.108.190:34935: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 08:42:39.001344 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 2208
2026-08-07T08:42:39Z chronyd exiting
I20260807 08:42:39.041344 29106 test_util.cc:175] -----------------------------------------------
I20260807 08:42:39.041460 29106 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786092133523677-29106-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (359 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 08:42:39.042484 29106 mini_oidc.cc:152] Starting JWKS server
I20260807 08:42:39.043242 29106 webserver.cc:460] Webserver started at https://127.0.0.1:34247/ using document root <none> and password file <none>
I20260807 08:42:39.043298 29106 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 08:42:39.043452 29106 webserver.cc:460] Webserver started at http://127.0.0.1:44129/ using document root <none> and password file <none>
2026-08-07T08:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:39Z Disabled control of system clock
I20260807 08:42:39.056161 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:35649
--webserver_interface=127.28.108.190
--webserver_port=0
--builtin_ntp_servers=127.28.108.148:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:35649
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34247/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/testchainca.cert with env {}
W20260807 08:42:39.152405 2317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:39.152666 2317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:39.152693 2317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:39.154667 2317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:39.154742 2317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:39.154757 2317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:39.154771 2317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:39.154786 2317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 08:42:39.154800 2317 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 08:42:39.154811 2317 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34247/jwks.json
I20260807 08:42:39.156750 2317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:35649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:35649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34247/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:39.157083 2317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:39.157347 2317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:39.163371 2325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:39.163408 2323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:39.163384 2322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:39.163769 2317 server_base.cc:1034] running on GCE node
I20260807 08:42:39.164072 2317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:39.164445 2317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:39.165737 2317 hybrid_clock.cc:648] HybridClock initialized: now 1786092159165695 us; error 51 us; skew 500 ppm
I20260807 08:42:39.167222 2317 webserver.cc:460] Webserver started at http://127.28.108.190:38217/ using document root <none> and password file <none>
I20260807 08:42:39.167582 2317 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:39.167667 2317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:39.167831 2317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:39.168998 2317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "81eba932b2e64654ad2c8b2c2df00be0"
format_stamp: "Formatted at 2026-08-07 08:42:39 on dist-test-slave-9zdj"
server_key: "fd0bfa03a2522aa2a4207c90c9b446f0"
server_key_iv: "16cc7b947d69d4f6c43c4da2c774ee7a"
server_key_version: "encryptionkey@0"
I20260807 08:42:39.169441 2317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "81eba932b2e64654ad2c8b2c2df00be0"
format_stamp: "Formatted at 2026-08-07 08:42:39 on dist-test-slave-9zdj"
server_key: "fd0bfa03a2522aa2a4207c90c9b446f0"
server_key_iv: "16cc7b947d69d4f6c43c4da2c774ee7a"
server_key_version: "encryptionkey@0"
I20260807 08:42:39.171056 2317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:39.172652 2331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:39.172991 2317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:39.173149 2317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "81eba932b2e64654ad2c8b2c2df00be0"
format_stamp: "Formatted at 2026-08-07 08:42:39 on dist-test-slave-9zdj"
server_key: "fd0bfa03a2522aa2a4207c90c9b446f0"
server_key_iv: "16cc7b947d69d4f6c43c4da2c774ee7a"
server_key_version: "encryptionkey@0"
I20260807 08:42:39.173245 2317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:39.195056 2317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:39.195437 2317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:39.195565 2317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:39.200430 2317 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:35649
I20260807 08:42:39.200522 2388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:35649 every 8 connection(s)
I20260807 08:42:39.200917 2317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:39.202244 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:39.204092 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 2317
I20260807 08:42:39.204298 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:39.204511 29106 external_mini_cluster.cc:1428] Setting key d721d029887800888e0a56bae39e6cda
I20260807 08:42:39.206058 2389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0: Bootstrap starting.
I20260807 08:42:39.206856 2389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:39.207227 2389 log.cc:824] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:39.208129 2389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0: No bootstrap required, opened a new log
I20260807 08:42:39.210775 2389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81eba932b2e64654ad2c8b2c2df00be0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35649 } }
I20260807 08:42:39.210990 2389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:39.211051 2389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81eba932b2e64654ad2c8b2c2df00be0, State: Initialized, Role: FOLLOWER
I20260807 08:42:39.211208 2389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81eba932b2e64654ad2c8b2c2df00be0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35649 } }
I20260807 08:42:39.211295 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:39.205171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.28.108.190:35649)
0807 08:42:39.205421 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:39.205430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:39.205460 (+ 30us) server_negotiation.cc:407] Connection header received
0807 08:42:39.205972 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:39.205980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:39.206052 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:39.206228 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:39.206471 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.207571 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.208523 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.209002 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.209156 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:39.210368 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:39.210373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:39.210374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:39.210577 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:39.210621 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:39.210714 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:39.210815 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:39.210896 (+ 81us) server_negotiation.cc:299] Negotiation successful
0807 08:42:39.211001 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":107,"threads_started":1}
I20260807 08:42:39.211333 2389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:39.211397 2389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:39.211473 2389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:39.212534 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:42:39.275709 2389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81eba932b2e64654ad2c8b2c2df00be0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35649 } }
I20260807 08:42:39.276104 2389 leader_election.cc:302] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81eba932b2e64654ad2c8b2c2df00be0; no voters:
I20260807 08:42:39.276428 2389 leader_election.cc:288] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:39.276825 2389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:39.276891 2394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:39.277109 2394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [term 1 LEADER]: Becoming Leader. State: Replica: 81eba932b2e64654ad2c8b2c2df00be0, State: Running, Role: LEADER
I20260807 08:42:39.277467 2394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81eba932b2e64654ad2c8b2c2df00be0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35649 } }
I20260807 08:42:39.278061 2396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81eba932b2e64654ad2c8b2c2df00be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81eba932b2e64654ad2c8b2c2df00be0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35649 } } }
I20260807 08:42:39.278158 2396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:39.278390 2395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81eba932b2e64654ad2c8b2c2df00be0. Latest consensus state: current_term: 1 leader_uuid: "81eba932b2e64654ad2c8b2c2df00be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81eba932b2e64654ad2c8b2c2df00be0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35649 } } }
I20260807 08:42:39.278785 2395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0 [sys.catalog]: This master's current role is: LEADER
W20260807 08:42:39.279505 2411 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:42:39.279590 2411 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:42:39.279688 2405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:39.279871 2405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:39.282009 2405 catalog_manager.cc:1347] Generated new cluster ID: 6d75041f82f141278db1e4ef2c954baf
I20260807 08:42:39.282078 2405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:39.295673 2405 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:39.296376 2405 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:39.302333 2405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81eba932b2e64654ad2c8b2c2df00be0: Generated new TSK 0
I20260807 08:42:39.302634 2405 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 08:42:39.331125 2422 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 08:42:39.331247 2422 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 08:42:39.331641 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:39.313984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.28.108.190:35649)
0807 08:42:39.314071 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:39.314073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:39.314087 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:42:39.314139 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:39.314140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:39.314152 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:39.314241 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0807 08:42:39.314473 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.315087 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.316316 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.316586 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.316663 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:39.316913 (+ 250us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 08:42:39.331409 (+ 14496us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 08:42:39.331579 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260807 08:42:39.331732 2392 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260807 08:42:39.331842 2421 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:39.313830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35649 (local address 127.0.0.1:37918)
0807 08:42:39.313996 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:39.314007 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:39.314062 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:39.314272 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:39.314273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:39.314275 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0807 08:42:39.314349 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:39.314351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.315192 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.315194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:39.316187 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:39.316191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.316698 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.316699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:39.316769 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:39.316777 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 08:42:39.331550 (+ 14773us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 08:42:39.331729 (+ 179us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.108.190:35649: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260807 08:42:39.332211 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 2317
2026-08-07T08:42:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 08:42:39.375114 29106 mini_oidc.cc:152] Starting JWKS server
I20260807 08:42:39.375991 29106 webserver.cc:460] Webserver started at https://127.0.0.1:37801/ using document root <none> and password file <none>
I20260807 08:42:39.376108 29106 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 08:42:39.376314 29106 webserver.cc:460] Webserver started at http://127.0.0.1:42869/ using document root <none> and password file <none>
2026-08-07T08:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:39Z Disabled control of system clock
I20260807 08:42:39.387771 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:33053
--webserver_interface=127.28.108.190
--webserver_port=0
--builtin_ntp_servers=127.28.108.148:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:33053
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37801/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 08:42:39.483733 2429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:39.483989 2429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:39.484015 2429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:39.485381 2429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:39.485445 2429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:39.485458 2429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:39.485471 2429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:39.485484 2429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 08:42:39.485496 2429 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 08:42:39.485508 2429 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37801/jwks.json
I20260807 08:42:39.487257 2429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:33053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:33053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37801/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:39.487609 2429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:39.487994 2429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:39.493918 2435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:39.493969 2436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:39.493983 2438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:39.494160 2429 server_base.cc:1034] running on GCE node
I20260807 08:42:39.494529 2429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:39.495051 2429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:39.496443 2429 hybrid_clock.cc:648] HybridClock initialized: now 1786092159496290 us; error 56 us; skew 500 ppm
I20260807 08:42:39.498297 2429 webserver.cc:460] Webserver started at http://127.28.108.190:41407/ using document root <none> and password file <none>
I20260807 08:42:39.498610 2429 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:39.498693 2429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:39.498865 2429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:39.500165 2429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "feb7b03bbefb466a92d05b75e83897c1"
format_stamp: "Formatted at 2026-08-07 08:42:39 on dist-test-slave-9zdj"
server_key: "2a53d84472ce2bbe50f918ddd90cfb40"
server_key_iv: "830d25d8044875277ddfe16259b083db"
server_key_version: "encryptionkey@0"
I20260807 08:42:39.500593 2429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "feb7b03bbefb466a92d05b75e83897c1"
format_stamp: "Formatted at 2026-08-07 08:42:39 on dist-test-slave-9zdj"
server_key: "2a53d84472ce2bbe50f918ddd90cfb40"
server_key_iv: "830d25d8044875277ddfe16259b083db"
server_key_version: "encryptionkey@0"
I20260807 08:42:39.502202 2429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260807 08:42:39.503203 2444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:39.503499 2429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:39.503626 2429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "feb7b03bbefb466a92d05b75e83897c1"
format_stamp: "Formatted at 2026-08-07 08:42:39 on dist-test-slave-9zdj"
server_key: "2a53d84472ce2bbe50f918ddd90cfb40"
server_key_iv: "830d25d8044875277ddfe16259b083db"
server_key_version: "encryptionkey@0"
I20260807 08:42:39.503719 2429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:39.518106 2429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:39.518718 2429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:39.518922 2429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:39.524287 2429 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:33053
I20260807 08:42:39.524361 2497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:33053 every 8 connection(s)
I20260807 08:42:39.524830 2429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:39.525234 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 2429
I20260807 08:42:39.525398 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:39.525810 29106 external_mini_cluster.cc:1428] Setting key 0079f26e58e401947ad332f7f326d16a
I20260807 08:42:39.525843 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:39.529963 2498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1: Bootstrap starting.
I20260807 08:42:39.530756 2498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:39.531112 2498 log.cc:824] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:39.532171 2498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1: No bootstrap required, opened a new log
I20260807 08:42:39.533843 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:39.526532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56458 (local address 127.28.108.190:33053)
0807 08:42:39.526804 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:39.526814 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:39.526844 (+ 30us) server_negotiation.cc:407] Connection header received
0807 08:42:39.527965 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:39.527976 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:39.528070 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:39.528279 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:39.528592 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.529873 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.530906 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.531583 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.531763 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:39.532796 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:39.532800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:39.532801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:39.532987 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:39.533023 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:39.533035 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:39.533096 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:39.533434 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 08:42:39.533565 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":120,"threads_started":1}
I20260807 08:42:39.535045 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:42:39.534904 2498 raft_consensus.cc:348] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb7b03bbefb466a92d05b75e83897c1" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33053 } }
I20260807 08:42:39.535108 2498 raft_consensus.cc:374] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:39.535144 2498 raft_consensus.cc:729] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb7b03bbefb466a92d05b75e83897c1, State: Initialized, Role: FOLLOWER
I20260807 08:42:39.535267 2498 consensus_queue.cc:260] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb7b03bbefb466a92d05b75e83897c1" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33053 } }
I20260807 08:42:39.535348 2498 raft_consensus.cc:388] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:39.535377 2498 raft_consensus.cc:482] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:39.535421 2498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:39.576695 2498 raft_consensus.cc:504] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb7b03bbefb466a92d05b75e83897c1" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33053 } }
I20260807 08:42:39.576900 2498 leader_election.cc:302] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feb7b03bbefb466a92d05b75e83897c1; no voters:
I20260807 08:42:39.577256 2498 leader_election.cc:288] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:39.577344 2503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:39.577549 2503 raft_consensus.cc:686] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [term 1 LEADER]: Becoming Leader. State: Replica: feb7b03bbefb466a92d05b75e83897c1, State: Running, Role: LEADER
I20260807 08:42:39.577684 2498 sys_catalog.cc:564] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:39.577726 2503 consensus_queue.cc:237] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb7b03bbefb466a92d05b75e83897c1" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33053 } }
I20260807 08:42:39.578253 2504 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feb7b03bbefb466a92d05b75e83897c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb7b03bbefb466a92d05b75e83897c1" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33053 } } }
I20260807 08:42:39.578359 2504 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:39.578640 2505 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feb7b03bbefb466a92d05b75e83897c1. Latest consensus state: current_term: 1 leader_uuid: "feb7b03bbefb466a92d05b75e83897c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb7b03bbefb466a92d05b75e83897c1" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33053 } } }
I20260807 08:42:39.578814 2505 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:39.579376 2509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:39.580014 2509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:39.582069 2509 catalog_manager.cc:1347] Generated new cluster ID: df4a83f7390f4c2caddc553cca1b8788
I20260807 08:42:39.582160 2509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:39.603121 2509 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:39.603847 2509 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:39.608121 2509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feb7b03bbefb466a92d05b75e83897c1: Generated new TSK 0
I20260807 08:42:39.608466 2509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:42:39.623984 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:39.618868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.28.108.190:33053)
0807 08:42:39.618965 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:39.618967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:39.618978 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:42:39.619029 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:39.619030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:39.619042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:39.619119 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0807 08:42:39.619356 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.620304 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.621693 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.621989 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.622068 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:39.622357 (+ 289us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 08:42:39.623709 (+ 1352us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37801/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 08:42:39.623879 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56464: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37801/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":49}
W20260807 08:42:39.624069 2501 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56464: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37801/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260807 08:42:39.624408 2528 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:39.618689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33053 (local address 127.0.0.1:56464)
0807 08:42:39.618877 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:39.618889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:39.618948 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:39.619143 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:39.619144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:39.619146 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0807 08:42:39.619217 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:39.619220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.620418 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.620419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:39.621530 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:39.621534 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.622093 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.622094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:39.622167 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:39.622175 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 08:42:39.623842 (+ 1667us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37801/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 08:42:39.623979 (+ 137us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.108.190:33053: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37801/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260807 08:42:39.624712 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 2429
E20260807 08:42:39.640949 2529 webserver.cc:533] Webserver: sslize failed:
E20260807 08:42:39.641081 2529 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-07T08:42:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (293 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T08:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:39Z Disabled control of system clock
I20260807 08:42:39.681066 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:44385
--webserver_interface=127.28.108.190
--webserver_port=0
--builtin_ntp_servers=127.28.108.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:44385
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/testchainca.cert with env {}
W20260807 08:42:39.771903 2534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:39.772148 2534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:39.772174 2534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:39.773682 2534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:39.773751 2534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:39.773765 2534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:39.773779 2534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:39.773792 2534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 08:42:39.773805 2534 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 08:42:39.773818 2534 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 08:42:39.775632 2534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:44385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:39.775991 2534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:39.776305 2534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:39.782227 2543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:39.782239 2540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:39.782229 2541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:39.782361 2534 server_base.cc:1034] running on GCE node
I20260807 08:42:39.782877 2534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:39.783191 2534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:39.784369 2534 hybrid_clock.cc:648] HybridClock initialized: now 1786092159784325 us; error 59 us; skew 500 ppm
I20260807 08:42:39.786306 2534 webserver.cc:460] Webserver started at http://127.28.108.190:38069/ using document root <none> and password file <none>
I20260807 08:42:39.786659 2534 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:39.786747 2534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:39.786860 2534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:39.788065 2534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "50db7abd3f594a12b7b4e18a9070325a"
format_stamp: "Formatted at 2026-08-07 08:42:39 on dist-test-slave-9zdj"
server_key: "3424645e26e1504f585f26d5548fdd4f"
server_key_iv: "b4d927ac3267e36b5f00de23edf33c78"
server_key_version: "encryptionkey@0"
I20260807 08:42:39.788429 2534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "50db7abd3f594a12b7b4e18a9070325a"
format_stamp: "Formatted at 2026-08-07 08:42:39 on dist-test-slave-9zdj"
server_key: "3424645e26e1504f585f26d5548fdd4f"
server_key_iv: "b4d927ac3267e36b5f00de23edf33c78"
server_key_version: "encryptionkey@0"
I20260807 08:42:39.790180 2534 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 08:42:39.791250 2549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:39.791605 2534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:39.791765 2534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "50db7abd3f594a12b7b4e18a9070325a"
format_stamp: "Formatted at 2026-08-07 08:42:39 on dist-test-slave-9zdj"
server_key: "3424645e26e1504f585f26d5548fdd4f"
server_key_iv: "b4d927ac3267e36b5f00de23edf33c78"
server_key_version: "encryptionkey@0"
I20260807 08:42:39.791894 2534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:39.801259 2534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:39.801859 2534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:39.802064 2534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:39.807112 2534 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:44385
I20260807 08:42:39.807155 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:44385 every 8 connection(s)
I20260807 08:42:39.807590 2534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:39.808799 2607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:39.811623 2607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a: Bootstrap starting.
I20260807 08:42:39.812369 2607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:39.812719 2607 log.cc:824] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:39.813707 2607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a: No bootstrap required, opened a new log
I20260807 08:42:39.815580 2607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50db7abd3f594a12b7b4e18a9070325a" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44385 } }
I20260807 08:42:39.815750 2607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:39.815774 2607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50db7abd3f594a12b7b4e18a9070325a, State: Initialized, Role: FOLLOWER
I20260807 08:42:39.815968 2607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50db7abd3f594a12b7b4e18a9070325a" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44385 } }
I20260807 08:42:39.816039 2607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:39.816085 2607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:39.816140 2607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:39.817149 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 2534
I20260807 08:42:39.817283 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:39.817448 29106 external_mini_cluster.cc:1428] Setting key 1e0e4e740ccb7a6572750cff7ea5f765
I20260807 08:42:39.824219 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:39.818051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.28.108.190:44385)
0807 08:42:39.818308 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:39.818317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:39.818349 (+ 32us) server_negotiation.cc:407] Connection header received
0807 08:42:39.818743 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:39.818751 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:39.818822 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:39.819031 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:39.819258 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.820224 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.821164 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.821728 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.821878 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:39.823251 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:39.823256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:39.823256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:42:39.823479 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:39.823528 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:42:39.823541 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:39.823652 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:39.823718 (+ 66us) server_negotiation.cc:299] Negotiation successful
0807 08:42:39.823889 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
I20260807 08:42:39.826056 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:42:39.870896 2607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50db7abd3f594a12b7b4e18a9070325a" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44385 } }
I20260807 08:42:39.871152 2607 leader_election.cc:302] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50db7abd3f594a12b7b4e18a9070325a; no voters:
I20260807 08:42:39.871490 2607 leader_election.cc:288] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:39.871611 2614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:39.871847 2614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [term 1 LEADER]: Becoming Leader. State: Replica: 50db7abd3f594a12b7b4e18a9070325a, State: Running, Role: LEADER
I20260807 08:42:39.871927 2607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:39.871966 2614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50db7abd3f594a12b7b4e18a9070325a" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44385 } }
I20260807 08:42:39.872511 2615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50db7abd3f594a12b7b4e18a9070325a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50db7abd3f594a12b7b4e18a9070325a" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44385 } } }
I20260807 08:42:39.872601 2615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:39.872862 2616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50db7abd3f594a12b7b4e18a9070325a. Latest consensus state: current_term: 1 leader_uuid: "50db7abd3f594a12b7b4e18a9070325a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50db7abd3f594a12b7b4e18a9070325a" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 44385 } } }
I20260807 08:42:39.872985 2616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:39.873464 2619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:39.873996 2619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:39.876117 2619 catalog_manager.cc:1347] Generated new cluster ID: 72ed209ca72b498195a0f6b705e38f70
I20260807 08:42:39.876186 2619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:39.909181 2619 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:39.909852 2619 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:39.913049 2619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50db7abd3f594a12b7b4e18a9070325a: Generated new TSK 0
I20260807 08:42:39.913287 2619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:42:41.221218 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:39.922005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.28.108.190:44385)
0807 08:42:39.922080 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:39.922082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:39.922205 (+ 123us) server_negotiation.cc:407] Connection header received
0807 08:42:39.922287 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:39.922289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:39.922302 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:39.922398 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0807 08:42:39.922560 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.923391 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.924364 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.924705 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.924783 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:39.925081 (+ 298us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 08:42:41.220919 (+1295838us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 08:42:41.221135 (+ 216us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54752: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":46}
W20260807 08:42:41.221328 2613 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54752: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260807 08:42:41.221711 2640 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:39.921964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:44385 (local address 127.0.0.1:54752)
0807 08:42:39.922116 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:39.922136 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:39.922207 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:39.922416 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:39.922417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:39.922419 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0807 08:42:39.922483 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:39.922485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.923503 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.923504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:39.924255 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:39.924260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:39.924829 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:39.924830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:39.924914 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:39.924921 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 08:42:41.221151 (+1296230us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 08:42:41.221495 (+ 344us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.108.190:44385: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260807 08:42:41.222373 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 2534
2026-08-07T08:42:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1575 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (105 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (98 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:41 dist-test-slave-9zdj krb5kdc[2653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:41 dist-test-slave-9zdj krb5kdc[2653](info): set up 2 sockets
Aug 07 08:42:41 dist-test-slave-9zdj krb5kdc[2653](info): commencing operation
krb5kdc: starting...
W20260807 08:42:42.642650 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:42 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:42.793888 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:36877
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:36877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:42.883559 2667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:42.883858 2667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:42.883903 2667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:42.885401 2667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:42.885495 2667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:42.885519 2667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:42.885540 2667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:42.885568 2667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:42.887336 2667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:36877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:36877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:42.887669 2667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:42.887934 2667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:42.894068 2672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:42.894073 2677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:42.894068 2673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:42.894529 2667 server_base.cc:1034] running on GCE node
I20260807 08:42:42.894847 2667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:42.895298 2667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:42.896525 2667 hybrid_clock.cc:648] HybridClock initialized: now 1786092162896469 us; error 79 us; skew 500 ppm
Aug 07 08:42:42 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092162, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:42.903966 2667 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:42.904590 2667 webserver.cc:460] Webserver started at http://127.28.108.190:32965/ using document root <none> and password file <none>
I20260807 08:42:42.904888 2667 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:42.904978 2667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:42.905181 2667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:42.906615 2667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "2e54af6a15cc447a81aa13c4fb4b85f7"
format_stamp: "Formatted at 2026-08-07 08:42:42 on dist-test-slave-9zdj"
server_key: "37a7903cf1acdbab2e28b22d60f086f9"
server_key_iv: "a07fdacf2d1c5791b980084a018e7090"
server_key_version: "encryptionkey@0"
I20260807 08:42:42.907052 2667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "2e54af6a15cc447a81aa13c4fb4b85f7"
format_stamp: "Formatted at 2026-08-07 08:42:42 on dist-test-slave-9zdj"
server_key: "37a7903cf1acdbab2e28b22d60f086f9"
server_key_iv: "a07fdacf2d1c5791b980084a018e7090"
server_key_version: "encryptionkey@0"
I20260807 08:42:42.908703 2667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:42:42.909790 2684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:42.910040 2667 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:42.910166 2667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "2e54af6a15cc447a81aa13c4fb4b85f7"
format_stamp: "Formatted at 2026-08-07 08:42:42 on dist-test-slave-9zdj"
server_key: "37a7903cf1acdbab2e28b22d60f086f9"
server_key_iv: "a07fdacf2d1c5791b980084a018e7090"
server_key_version: "encryptionkey@0"
I20260807 08:42:42.910270 2667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:42.932865 2667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:42.935636 2667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:42.935863 2667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:42.940922 2667 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:36877
I20260807 08:42:42.940963 2738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:36877 every 8 connection(s)
I20260807 08:42:42.941475 2667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:42.942725 2739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:42.945338 2739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7: Bootstrap starting.
I20260807 08:42:42.946018 2739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:42.946367 2739 log.cc:824] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:42.947144 2739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7: No bootstrap required, opened a new log
I20260807 08:42:42.948944 2739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54af6a15cc447a81aa13c4fb4b85f7" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36877 } }
I20260807 08:42:42.949126 2739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:42.949177 2739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e54af6a15cc447a81aa13c4fb4b85f7, State: Initialized, Role: FOLLOWER
I20260807 08:42:42.949365 2739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54af6a15cc447a81aa13c4fb4b85f7" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36877 } }
I20260807 08:42:42.949456 2739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:42.949505 2739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:42.949564 2739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:42.950184 2739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54af6a15cc447a81aa13c4fb4b85f7" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36877 } }
I20260807 08:42:42.950321 2739 leader_election.cc:302] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e54af6a15cc447a81aa13c4fb4b85f7; no voters:
I20260807 08:42:42.950542 2739 leader_election.cc:288] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:42.950636 2742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:42.950846 2739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:42.951094 2742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [term 1 LEADER]: Becoming Leader. State: Replica: 2e54af6a15cc447a81aa13c4fb4b85f7, State: Running, Role: LEADER
I20260807 08:42:42.951225 2742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54af6a15cc447a81aa13c4fb4b85f7" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36877 } }
I20260807 08:42:42.951583 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 2667
I20260807 08:42:42.951712 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:42.951756 2745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e54af6a15cc447a81aa13c4fb4b85f7. Latest consensus state: current_term: 1 leader_uuid: "2e54af6a15cc447a81aa13c4fb4b85f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54af6a15cc447a81aa13c4fb4b85f7" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36877 } } }
I20260807 08:42:42.951946 29106 external_mini_cluster.cc:1428] Setting key 1d8dba16db86f181040298074adaacd3
I20260807 08:42:42.951859 2745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:42.952646 2753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:42.953063 2744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e54af6a15cc447a81aa13c4fb4b85f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54af6a15cc447a81aa13c4fb4b85f7" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 36877 } } }
I20260807 08:42:42.953151 2744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:42.953296 2753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:42.955288 2753 catalog_manager.cc:1347] Generated new cluster ID: 4e5f74f6e6d2436b8d9f34eecc34f82e
I20260807 08:42:42.955353 2753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:42:42 dist-test-slave-9zdj krb5kdc[2653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:42.966035 2753 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:42.966058 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:42.952590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.28.108.190:36877)
0807 08:42:42.952827 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:42.952840 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:42.953053 (+ 213us) server_negotiation.cc:407] Connection header received
0807 08:42:42.953221 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:42.953230 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:42.953303 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:42.953465 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:42.954333 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:42.955820 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:42.957083 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:42.957806 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:42.957922 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:42.961433 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:42.961439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:42.961440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:42.961455 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:42.963926 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:42.964460 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:42.964464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:42.964465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:42.964595 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:42.964856 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:42.964857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:42.964858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:42.965210 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:42.965378 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:42.965574 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 08:42:42.965767 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260807 08:42:42.966802 2753 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:42.973024 2753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e54af6a15cc447a81aa13c4fb4b85f7: Generated new TSK 0
I20260807 08:42:42.973356 2753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:42:43.019259 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:36877
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:43.153578 2767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:43.153957 2767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:43.154008 2767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:43.155534 2767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:43.155637 2767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:43.155706 2767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:43.157716 2767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:36877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:43.158035 2767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:43.158324 2767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:43.164577 2776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:43.164541 2774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:43.164541 2773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:43.164834 2767 server_base.cc:1034] running on GCE node
I20260807 08:42:43.165158 2767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:43.165480 2767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:43.166682 2767 hybrid_clock.cc:648] HybridClock initialized: now 1786092163166662 us; error 60 us; skew 500 ppm
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:43.169936 2767 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:42:43.170519 2767 webserver.cc:460] Webserver started at http://127.28.108.129:41019/ using document root <none> and password file <none>
I20260807 08:42:43.170799 2767 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:43.170883 2767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:43.171058 2767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:43.172386 2767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "2ce8b3e514db47dfb3ecfaa7b43fcb85"
format_stamp: "Formatted at 2026-08-07 08:42:43 on dist-test-slave-9zdj"
server_key: "99b25444a73cbce8a8bbf9c828d89920"
server_key_iv: "d251336f52047a28db3ed71dda931b8f"
server_key_version: "encryptionkey@0"
I20260807 08:42:43.172813 2767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "2ce8b3e514db47dfb3ecfaa7b43fcb85"
format_stamp: "Formatted at 2026-08-07 08:42:43 on dist-test-slave-9zdj"
server_key: "99b25444a73cbce8a8bbf9c828d89920"
server_key_iv: "d251336f52047a28db3ed71dda931b8f"
server_key_version: "encryptionkey@0"
I20260807 08:42:43.174638 2767 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 08:42:43.175710 2783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:43.175992 2767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:43.176121 2767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "2ce8b3e514db47dfb3ecfaa7b43fcb85"
format_stamp: "Formatted at 2026-08-07 08:42:43 on dist-test-slave-9zdj"
server_key: "99b25444a73cbce8a8bbf9c828d89920"
server_key_iv: "d251336f52047a28db3ed71dda931b8f"
server_key_version: "encryptionkey@0"
I20260807 08:42:43.176210 2767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:43.202627 2767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:43.205837 2767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:43.206091 2767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:43.206408 2767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:43.206789 2767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:43.206837 2767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:43.206894 2767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:43.206916 2767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:43.214875 2767 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:35935
I20260807 08:42:43.214916 2905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:35935 every 8 connection(s)
I20260807 08:42:43.215363 2767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:42:43.218394 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 2767
I20260807 08:42:43.218569 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:43.218839 29106 external_mini_cluster.cc:1428] Setting key b3987e6e8d1696c28291d3e202f2b30a
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:43.228186 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:43.216530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:41603 (local address 127.28.108.190:36877)
0807 08:42:43.216702 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:43.216705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:43.216955 (+ 250us) server_negotiation.cc:407] Connection header received
0807 08:42:43.218164 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:43.218166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:43.218187 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:43.218420 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:43.219304 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.219877 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.221009 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.221249 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.221338 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:43.224236 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:43.224238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:43.224239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:43.224249 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:43.226302 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.226981 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.226983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.226984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.227060 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.227365 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.227366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.227367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.227522 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:43.227612 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:43.227956 (+ 344us) server_negotiation.cc:299] Negotiation successful
0807 08:42:43.228082 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260807 08:42:43.228466 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:43.216502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:36877 (local address 127.28.108.129:41603)
0807 08:42:43.216802 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:43.216848 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:43.218000 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:43.218536 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:43.218540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:43.218956 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:43.219194 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:43.219199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.220004 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.220005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:43.220900 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:43.220903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.221349 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.221350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:43.221538 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:43.222218 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:43.222230 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:43.224092 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:43.226509 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:43.226523 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:43.226527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:43.226877 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:43.227139 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:43.227140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:43.227140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:43.227284 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:43.227602 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:43.227612 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:43.227786 (+ 174us) client_negotiation.cc:769] Sending connection context
0807 08:42:43.227920 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 08:42:43.228121 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
I20260807 08:42:43.229059 2907 heartbeater.cc:347] Connected to a master server at 127.28.108.190:36877
I20260807 08:42:43.229199 2907 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:43.229449 2907 heartbeater.cc:511] Master 127.28.108.190:36877 requested a full tablet report, sending...
I20260807 08:42:43.230363 2701 ts_manager.cc:194] Registered new tserver with Master: 2ce8b3e514db47dfb3ecfaa7b43fcb85 (127.28.108.129:35935)
I20260807 08:42:43.231079 2701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:41603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:42:43.254434 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:36877
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:43.347492 2912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:43.347842 2912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:43.347893 2912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:43.349298 2912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:43.349395 2912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:43.349440 2912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:43.351446 2912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:36877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:43.351783 2912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:43.352097 2912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:43.357859 2920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:43.357877 2922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:43.357877 2919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:43.358633 2912 server_base.cc:1034] running on GCE node
I20260807 08:42:43.358846 2912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:43.359083 2912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:43.360239 2912 hybrid_clock.cc:648] HybridClock initialized: now 1786092163360194 us; error 65 us; skew 500 ppm
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:43.363255 2912 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:42:43.363806 2912 webserver.cc:460] Webserver started at http://127.28.108.130:42711/ using document root <none> and password file <none>
I20260807 08:42:43.364064 2912 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:43.364122 2912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:43.364279 2912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:43.365456 2912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "5b41befe75884d4ca0ad7f0ef85bc285"
format_stamp: "Formatted at 2026-08-07 08:42:43 on dist-test-slave-9zdj"
server_key: "e68a7134a712839f2d3209bc34eceafa"
server_key_iv: "b76122d80468f4f938bebc5681031371"
server_key_version: "encryptionkey@0"
I20260807 08:42:43.365932 2912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "5b41befe75884d4ca0ad7f0ef85bc285"
format_stamp: "Formatted at 2026-08-07 08:42:43 on dist-test-slave-9zdj"
server_key: "e68a7134a712839f2d3209bc34eceafa"
server_key_iv: "b76122d80468f4f938bebc5681031371"
server_key_version: "encryptionkey@0"
I20260807 08:42:43.367472 2912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:43.368486 2929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:43.368861 2912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:43.368956 2912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "5b41befe75884d4ca0ad7f0ef85bc285"
format_stamp: "Formatted at 2026-08-07 08:42:43 on dist-test-slave-9zdj"
server_key: "e68a7134a712839f2d3209bc34eceafa"
server_key_iv: "b76122d80468f4f938bebc5681031371"
server_key_version: "encryptionkey@0"
I20260807 08:42:43.369086 2912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:43.394029 2912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:43.397330 2912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:43.397475 2912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:43.397891 2912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:43.398274 2912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:43.398475 2912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:43.398535 2912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:43.398563 2912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:43.406296 2912 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:37525
I20260807 08:42:43.406334 3049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:37525 every 8 connection(s)
I20260807 08:42:43.406733 2912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:43.414189 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 2912
I20260807 08:42:43.414343 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:43.414559 29106 external_mini_cluster.cc:1428] Setting key cca05b1e8d38a9b5071823961ec6c0d0
I20260807 08:42:43.418699 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:43.407633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:58247 (local address 127.28.108.190:36877)
0807 08:42:43.407735 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:43.407738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:43.407952 (+ 214us) server_negotiation.cc:407] Connection header received
0807 08:42:43.408953 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:43.408955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:43.408968 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:43.409057 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:43.410040 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.410599 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.411853 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.412095 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.412173 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:43.415292 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:43.415294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:43.415294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:43.415303 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:43.416846 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.417387 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.417388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.417389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.417465 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.417859 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.417861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.417861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.418016 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:43.418088 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:43.418496 (+ 408us) server_negotiation.cc:299] Negotiation successful
0807 08:42:43.418597 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 08:42:43.418946 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:43.407590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:36877 (local address 127.28.108.130:58247)
0807 08:42:43.407765 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:43.407811 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:43.408825 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:43.409231 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:43.409235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:43.409579 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:43.409911 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:43.409917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.410719 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.410720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:43.411746 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:43.411748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.412232 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.412233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:43.412372 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:43.412929 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:43.412938 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:43.415052 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:43.416994 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:43.417005 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:43.417008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:43.417297 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:43.417552 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:43.417554 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:43.417554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:43.417766 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:43.418097 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:43.418104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:43.418318 (+ 214us) client_negotiation.cc:769] Sending connection context
0807 08:42:43.418441 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 08:42:43.418652 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260807 08:42:43.419577 3050 heartbeater.cc:347] Connected to a master server at 127.28.108.190:36877
I20260807 08:42:43.419718 3050 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:43.419957 3050 heartbeater.cc:511] Master 127.28.108.190:36877 requested a full tablet report, sending...
I20260807 08:42:43.420456 2701 ts_manager.cc:194] Registered new tserver with Master: 5b41befe75884d4ca0ad7f0ef85bc285 (127.28.108.130:37525)
I20260807 08:42:43.420936 2701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:58247
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:42:43.453821 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:36877
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:43.549415 3055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:43.549768 3055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:43.549808 3055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:43.551311 3055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:43.551370 3055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:43.551401 3055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:43.554070 3055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:36877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:43.554531 3055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:43.554899 3055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:43.562109 3063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:43.562088 3061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:43.562088 3060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:43.562536 3055 server_base.cc:1034] running on GCE node
I20260807 08:42:43.562886 3055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:43.563215 3055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:43.564486 3055 hybrid_clock.cc:648] HybridClock initialized: now 1786092163564444 us; error 54 us; skew 500 ppm
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:43.568146 3055 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:42:43.568722 3055 webserver.cc:460] Webserver started at http://127.28.108.131:42379/ using document root <none> and password file <none>
I20260807 08:42:43.569011 3055 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:43.569097 3055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:43.569265 3055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:43.570508 3055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "b5165ce1fe5e4de89a3761a3627dca39"
format_stamp: "Formatted at 2026-08-07 08:42:43 on dist-test-slave-9zdj"
server_key: "59648cbc75c5955d53ef06d0e9dd31e8"
server_key_iv: "77a62a646ed7da1170ccabfea6ba4797"
server_key_version: "encryptionkey@0"
I20260807 08:42:43.570906 3055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "b5165ce1fe5e4de89a3761a3627dca39"
format_stamp: "Formatted at 2026-08-07 08:42:43 on dist-test-slave-9zdj"
server_key: "59648cbc75c5955d53ef06d0e9dd31e8"
server_key_iv: "77a62a646ed7da1170ccabfea6ba4797"
server_key_version: "encryptionkey@0"
I20260807 08:42:43.572484 3055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:42:43.573514 3071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:43.573828 3055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:43.573949 3055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "b5165ce1fe5e4de89a3761a3627dca39"
format_stamp: "Formatted at 2026-08-07 08:42:43 on dist-test-slave-9zdj"
server_key: "59648cbc75c5955d53ef06d0e9dd31e8"
server_key_iv: "77a62a646ed7da1170ccabfea6ba4797"
server_key_version: "encryptionkey@0"
I20260807 08:42:43.574033 3055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:43.595713 3055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:43.599629 3055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:43.599896 3055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:43.600216 3055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:43.600610 3055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:43.600675 3055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:43.600713 3055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:43.600730 3055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:43.607496 3055 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:33377
I20260807 08:42:43.607689 3191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:33377 every 8 connection(s)
I20260807 08:42:43.607990 3055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:43.611287 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 3055
I20260807 08:42:43.611449 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:43.611667 29106 external_mini_cluster.cc:1428] Setting key 734ea6965fefbf7779c52cfac3f71bc2
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:43.619762 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:43.608956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:49387 (local address 127.28.108.190:36877)
0807 08:42:43.609028 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:43.609031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:43.609206 (+ 175us) server_negotiation.cc:407] Connection header received
0807 08:42:43.610254 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:43.610255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:43.610268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:43.610351 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:43.611499 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.612415 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.613678 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.613918 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.613983 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:43.616541 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:43.616542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:43.616543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:43.616551 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:43.618248 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.618739 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.618740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.618740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.618806 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.619059 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.619060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.619060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.619188 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:43.619247 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:43.619610 (+ 363us) server_negotiation.cc:299] Negotiation successful
0807 08:42:43.619688 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 08:42:43.620100 3194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:43.608867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:36877 (local address 127.28.108.131:49387)
0807 08:42:43.609036 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:43.609083 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:43.610135 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:43.610529 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:43.610535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:43.611011 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:43.611363 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:43.611370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.612538 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.612539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:43.613541 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:43.613544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.614005 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.614006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:43.614133 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:43.614850 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:43.614859 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:43.616432 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:43.618363 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:43.618372 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:43.618374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:43.618650 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:43.618870 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:43.618871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:43.618871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:43.618992 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:43.619311 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:43.619318 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:43.619470 (+ 152us) client_negotiation.cc:769] Sending connection context
0807 08:42:43.619572 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 08:42:43.619792 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260807 08:42:43.620767 3192 heartbeater.cc:347] Connected to a master server at 127.28.108.190:36877
I20260807 08:42:43.620899 3192 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:43.621110 3192 heartbeater.cc:511] Master 127.28.108.190:36877 requested a full tablet report, sending...
I20260807 08:42:43.621703 2701 ts_manager.cc:194] Registered new tserver with Master: b5165ce1fe5e4de89a3761a3627dca39 (127.28.108.131:33377)
I20260807 08:42:43.622095 2701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:49387
I20260807 08:42:43.623580 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/test-admin.keytab.
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:43.643743 29106 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 08:42:43.645776 3199 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:43.644914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.28.108.129:35935)
0807 08:42:43.645093 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:43.645099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:43.645112 (+ 13us) server_negotiation.cc:407] Connection header received
0807 08:42:43.645158 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:43.645160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:43.645175 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:43.645316 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:43.645676 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40856: BlockingRecv error: recv got EOF from 127.0.0.1:40856 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
W20260807 08:42:43.646014 29106 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5cc)
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:43.647217 29106 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:43.648219 29106 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
I20260807 08:42:43.658609 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:43.648896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.28.108.129:35935)
0807 08:42:43.648968 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:43.648970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:43.648984 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:42:43.649100 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:43.649101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:43.649115 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:43.649193 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:43.649988 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.650958 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.651851 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.652287 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.652375 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:43.655247 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:43.655265 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:43.655265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:43.655274 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:43.657144 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.657578 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.657587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.657588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.657720 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.657968 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.657969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.657969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.658149 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:43.658345 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:43.658394 (+ 49us) server_negotiation.cc:299] Negotiation successful
0807 08:42:43.658488 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260807 08:42:43.661157 3199 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:43.660441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40862 (local address 127.28.108.129:35935)
0807 08:42:43.660480 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:43.660482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:43.660551 (+ 69us) server_negotiation.cc:407] Connection header received
0807 08:42:43.660590 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:43.660591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:43.660600 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:43.660651 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:43.661085 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40862: BlockingRecv error: recv got EOF from 127.0.0.1:40862 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260807 08:42:43.661330 29106 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5cc' of type FILE
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:43.662722 29106 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:43.663744 29106 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
I20260807 08:42:43.672883 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:43.664379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.28.108.129:35935)
0807 08:42:43.664473 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:43.664475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:43.664572 (+ 97us) server_negotiation.cc:407] Connection header received
0807 08:42:43.664672 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:43.664673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:43.664685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:43.664756 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:43.665191 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.665873 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.666600 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.666815 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.666896 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:43.669409 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:43.669410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:43.669411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:43.669419 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:43.671311 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.671783 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.671784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.671785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.671848 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.672247 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.672248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.672249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.672380 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:43.672470 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:43.672646 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 08:42:43.672790 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260807 08:42:43.674991 3199 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:43.674162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.28.108.129:35935)
0807 08:42:43.674215 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:43.674217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:43.674226 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:42:43.674362 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:43.674362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:43.674372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:43.674438 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:43.674925 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40884: BlockingRecv error: recv got EOF from 127.0.0.1:40884 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260807 08:42:43.675004 29106 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786092133523677-29106-0/krb5kdc/krb5cc' of type FILE
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:43.676221 29106 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:43.677201 29106 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[2653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
I20260807 08:42:43.685667 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:43.677947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.28.108.129:35935)
0807 08:42:43.677996 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:43.677999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:43.678085 (+ 86us) server_negotiation.cc:407] Connection header received
0807 08:42:43.678144 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:43.678145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:43.678154 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:43.678194 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:43.678628 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.679136 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.679946 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:43.680186 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:43.680248 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:43.682873 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:43.682875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:43.682875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:43.682883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:43.684524 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.684933 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.684934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.684935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.684988 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:43.685186 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:43.685187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:43.685187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:43.685285 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:43.685359 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:43.685471 (+ 112us) server_negotiation.cc:299] Negotiation successful
0807 08:42:43.685536 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:42:43.686404 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 2767
I20260807 08:42:43.694408 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 2912
I20260807 08:42:43.701087 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 3055
I20260807 08:42:43.709707 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 2667
2026-08-07T08:42:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2285 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[3213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[3213](info): set up 2 sockets
Aug 07 08:42:43 dist-test-slave-9zdj krb5kdc[3213](info): commencing operation
krb5kdc: starting...
W20260807 08:42:44.860507 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:44 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.190.keytab.
Entry for principal oryx/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.190.keytab.
I20260807 08:42:45.006675 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:33641
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.190.keytab
--principal=oryx/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:33641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:45.103672 3227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:45.103920 3227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:45.103948 3227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:45.105269 3227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:45.105322 3227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:45.105337 3227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:45.105350 3227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:45.105363 3227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:45.107095 3227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:33641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.190.keytab
--principal=oryx/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:33641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:45.107393 3227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:45.107682 3227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:45.113737 3233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:45.114070 3235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:45.114182 3227 server_base.cc:1034] running on GCE node
W20260807 08:42:45.114126 3232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:45.114569 3227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:45.115011 3227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:45.116240 3227 hybrid_clock.cc:648] HybridClock initialized: now 1786092165116200 us; error 71 us; skew 500 ppm
Aug 07 08:42:45 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:45.119321 3227 init.cc:377] Logged in from keytab as oryx/127.28.108.190@KRBTEST.COM (short username oryx)
I20260807 08:42:45.119872 3227 webserver.cc:460] Webserver started at http://127.28.108.190:35065/ using document root <none> and password file <none>
I20260807 08:42:45.120152 3227 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:45.120227 3227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:45.120389 3227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:45.121712 3227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "ed8bac22a00c4500bdc8eeebc76d77a0"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "de52b14b7e65c2db5e6bff5e3de87879"
server_key_iv: "0b491edecea2e13bb42b08f23858f24b"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.122133 3227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "ed8bac22a00c4500bdc8eeebc76d77a0"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "de52b14b7e65c2db5e6bff5e3de87879"
server_key_iv: "0b491edecea2e13bb42b08f23858f24b"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.123837 3227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:42:45.124804 3243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:45.125089 3227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:45.125231 3227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "ed8bac22a00c4500bdc8eeebc76d77a0"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "de52b14b7e65c2db5e6bff5e3de87879"
server_key_iv: "0b491edecea2e13bb42b08f23858f24b"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.125321 3227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:45.135798 3227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:45.138805 3227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:45.139020 3227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:45.144111 3227 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:33641
I20260807 08:42:45.144150 3297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:33641 every 8 connection(s)
I20260807 08:42:45.144568 3227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:45.145855 3298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:45.148404 3298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0: Bootstrap starting.
I20260807 08:42:45.149085 3298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:45.149420 3298 log.cc:824] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:45.150550 3298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0: No bootstrap required, opened a new log
I20260807 08:42:45.152504 3298 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } }
I20260807 08:42:45.152683 3298 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:45.152710 3298 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8bac22a00c4500bdc8eeebc76d77a0, State: Initialized, Role: FOLLOWER
I20260807 08:42:45.152901 3298 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } }
I20260807 08:42:45.152995 3298 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:45.153023 3298 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:45.153103 3298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:45.153713 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 3227
I20260807 08:42:45.153853 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:45.154070 29106 external_mini_cluster.cc:1428] Setting key f4789b61544fe8f17441d57417c25253
I20260807 08:42:45.154472 3298 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } }
I20260807 08:42:45.154606 3298 leader_election.cc:302] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed8bac22a00c4500bdc8eeebc76d77a0; no voters:
I20260807 08:42:45.154811 3298 leader_election.cc:288] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:45.154855 3304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:45.155015 3304 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 1 LEADER]: Becoming Leader. State: Replica: ed8bac22a00c4500bdc8eeebc76d77a0, State: Running, Role: LEADER
I20260807 08:42:45.155143 3298 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:45.155129 3304 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } }
I20260807 08:42:45.155638 3306 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed8bac22a00c4500bdc8eeebc76d77a0. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } }
I20260807 08:42:45.155730 3306 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:45.155900 3305 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } }
I20260807 08:42:45.155980 3305 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:45.156107 3310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:45.157164 3310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:45.160071 3310 catalog_manager.cc:1347] Generated new cluster ID: 172eadb7b96140c092e0d4aede3b8c81
I20260807 08:42:45.160141 3310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:42:45 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:45.164799 3310 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:45.165655 3310 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:45.168870 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:45.154660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.28.108.190:33641)
0807 08:42:45.154920 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:45.154933 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:45.154982 (+ 49us) server_negotiation.cc:407] Connection header received
0807 08:42:45.155387 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:45.155396 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:45.155465 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:45.155624 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:45.156888 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.158420 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.159675 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.160386 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.160525 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:45.163836 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:45.163841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:45.163842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:45.163857 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:45.166921 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:45.167453 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:45.167457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:45.167457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:45.167579 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:45.167831 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:45.167832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:45.167832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:45.168120 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:45.168286 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:45.168442 (+ 156us) server_negotiation.cc:299] Negotiation successful
0807 08:42:45.168595 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
I20260807 08:42:45.170056 3310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0: Generated new TSK 0
I20260807 08:42:45.170264 3310 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab.
Entry for principal oryx/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab.
I20260807 08:42:45.211735 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33641
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab
--principal=oryx/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:45.307783 3325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:45.308194 3325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:45.308251 3325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:45.309533 3325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:45.309610 3325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:45.309684 3325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:45.311275 3325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab
--principal=oryx/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:45.311581 3325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:45.311846 3325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:45.317759 3333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:45.317759 3335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:45.317764 3332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:45.318068 3325 server_base.cc:1034] running on GCE node
I20260807 08:42:45.318331 3325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:45.318620 3325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:45.319819 3325 hybrid_clock.cc:648] HybridClock initialized: now 1786092165319774 us; error 63 us; skew 500 ppm
Aug 07 08:42:45 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:45.322875 3325 init.cc:377] Logged in from keytab as oryx/127.28.108.129@KRBTEST.COM (short username oryx)
I20260807 08:42:45.323441 3325 webserver.cc:460] Webserver started at http://127.28.108.129:42139/ using document root <none> and password file <none>
I20260807 08:42:45.323709 3325 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:45.323787 3325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:45.323961 3325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:45.325213 3325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "ffee67b8ac02fca8315a37cb83f678ef"
server_key_iv: "57e53792a56b66a7348dc38d98760827"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.325732 3325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "ffee67b8ac02fca8315a37cb83f678ef"
server_key_iv: "57e53792a56b66a7348dc38d98760827"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.327318 3325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:45.328528 3342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:45.328840 3325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:45.328972 3325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "ffee67b8ac02fca8315a37cb83f678ef"
server_key_iv: "57e53792a56b66a7348dc38d98760827"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.329078 3325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:45.349299 3325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:45.352133 3325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:45.352429 3325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:45.352735 3325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:45.353091 3325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:45.353150 3325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:45.353197 3325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:45.353221 3325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:45.360394 3325 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:37387
I20260807 08:42:45.360446 3464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:37387 every 8 connection(s)
I20260807 08:42:45.360808 3325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
Aug 07 08:42:45 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:45.369813 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 3325
I20260807 08:42:45.369954 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:45.370220 29106 external_mini_cluster.cc:1428] Setting key d5c44d928628d6821b701de1a9dc52c5
I20260807 08:42:45.373016 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:45.361604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.129:33709)
0807 08:42:45.361814 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:45.361859 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:45.362368 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:45.362609 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:45.362612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:45.363289 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:45.363510 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:45.363515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.364401 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.364402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:45.365496 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:45.365499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.366001 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.366003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:45.366128 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:45.366986 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:45.366997 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:45.368831 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:45.370982 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:45.370995 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:45.370998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:45.371388 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:45.371763 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:45.371764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:45.371765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:45.371917 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:45.372344 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:45.372352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:45.372517 (+ 165us) client_negotiation.cc:769] Sending connection context
0807 08:42:45.372625 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 08:42:45.372810 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":108,"threads_started":1}
I20260807 08:42:45.373009 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:45.361675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:33709 (local address 127.28.108.190:33641)
0807 08:42:45.361771 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:45.361780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:45.361957 (+ 177us) server_negotiation.cc:407] Connection header received
0807 08:42:45.362467 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:45.362469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:45.362481 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:45.362532 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:45.363614 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.364141 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.365661 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.365909 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.365972 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:45.368969 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:45.368971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:45.368971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:45.368979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:45.370875 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:45.371497 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:45.371498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:45.371499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:45.371583 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:45.372003 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:45.372004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:45.372004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:45.372184 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:45.372264 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:45.372675 (+ 411us) server_negotiation.cc:299] Negotiation successful
0807 08:42:45.372861 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 08:42:45.373860 3465 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33641
I20260807 08:42:45.373976 3465 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:45.374233 3465 heartbeater.cc:511] Master 127.28.108.190:33641 requested a full tablet report, sending...
I20260807 08:42:45.374965 3260 ts_manager.cc:194] Registered new tserver with Master: 9ffc76d5ad2f42879ee70fbe8baf1cb3 (127.28.108.129:37387)
I20260807 08:42:45.375608 3260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:33709
WARNING: no policy specified for oryx/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab.
Entry for principal oryx/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab.
I20260807 08:42:45.405687 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33641
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab
--principal=oryx/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:45.495328 3471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:45.495582 3471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:45.495610 3471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:45.496956 3471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:45.497020 3471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:45.497052 3471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:45.498821 3471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab
--principal=oryx/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:45.499132 3471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:45.499449 3471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:42:45.505977 3471 server_base.cc:1034] running on GCE node
W20260807 08:42:45.506237 3478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:45.505978 3480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:45.505977 3477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:45.506774 3471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:45.507117 3471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:45.508440 3471 hybrid_clock.cc:648] HybridClock initialized: now 1786092165508383 us; error 71 us; skew 500 ppm
Aug 07 08:42:45 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:45.512245 3471 init.cc:377] Logged in from keytab as oryx/127.28.108.130@KRBTEST.COM (short username oryx)
I20260807 08:42:45.512957 3471 webserver.cc:460] Webserver started at http://127.28.108.130:37111/ using document root <none> and password file <none>
I20260807 08:42:45.513250 3471 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:45.513334 3471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:45.513517 3471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:45.514897 3471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "c58bb082ae12456686f0cd99fced750c"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "b99d218089f41897304af7cd574600fe"
server_key_iv: "8ca3237201af2a0279288ecc13a0c8d0"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.515377 3471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "c58bb082ae12456686f0cd99fced750c"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "b99d218089f41897304af7cd574600fe"
server_key_iv: "8ca3237201af2a0279288ecc13a0c8d0"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.517088 3471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:42:45.518438 3488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:45.518853 3471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:42:45.519021 3471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "c58bb082ae12456686f0cd99fced750c"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "b99d218089f41897304af7cd574600fe"
server_key_iv: "8ca3237201af2a0279288ecc13a0c8d0"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.519132 3471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:45.566658 3471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:45.570036 3471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:45.570237 3471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:45.570611 3471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:45.571040 3471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:45.571108 3471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:45.571158 3471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:45.571188 3471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:45.579046 3471 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:41927
I20260807 08:42:45.579092 3610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:41927 every 8 connection(s)
I20260807 08:42:45.579510 3471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:45.584079 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 3471
I20260807 08:42:45.584234 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:42:45.584471 29106 external_mini_cluster.cc:1428] Setting key 93b70baaa3de32bd1a60dde77d6c2ad4
Aug 07 08:42:45 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:45.593928 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:45.580530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:51691 (local address 127.28.108.190:33641)
0807 08:42:45.580632 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:45.580634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:45.580850 (+ 216us) server_negotiation.cc:407] Connection header received
0807 08:42:45.581431 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:45.581432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:45.581444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:45.581515 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:45.582734 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.583478 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.585460 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.585758 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.585873 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:45.589429 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:45.589431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:45.589432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:45.589442 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:45.591478 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:45.592051 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:45.592053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:45.592054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:45.592136 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:45.592521 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:45.592522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:45.592523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:45.592714 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:45.592801 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:45.593549 (+ 748us) server_negotiation.cc:299] Negotiation successful
0807 08:42:45.593764 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 08:42:45.594053 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:45.580433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.130:51691)
0807 08:42:45.580632 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:45.580684 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:45.581284 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:45.581579 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:45.581583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:45.582348 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:45.582589 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:45.582594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.583610 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.583612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:45.585190 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:45.585195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.585892 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.585894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:45.586056 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:45.586956 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:45.586967 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:45.589265 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:45.591608 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:45.591620 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:45.591623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:45.591954 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:45.592266 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:45.592267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:45.592267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:45.592421 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:45.592811 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:45.592819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:45.593098 (+ 279us) client_negotiation.cc:769] Sending connection context
0807 08:42:45.593505 (+ 407us) client_negotiation.cc:240] Negotiation successful
0807 08:42:45.593772 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
I20260807 08:42:45.594707 3611 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33641
I20260807 08:42:45.594856 3611 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:45.595103 3611 heartbeater.cc:511] Master 127.28.108.190:33641 requested a full tablet report, sending...
I20260807 08:42:45.595584 3260 ts_manager.cc:194] Registered new tserver with Master: c58bb082ae12456686f0cd99fced750c (127.28.108.130:41927)
I20260807 08:42:45.596073 3260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:51691
WARNING: no policy specified for oryx/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab.
Entry for principal oryx/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab.
I20260807 08:42:45.621992 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33641
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab
--principal=oryx/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:45.712962 3619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:45.713264 3619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:45.713316 3619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:45.714811 3619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:45.714876 3619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:45.714910 3619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:45.716562 3619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab
--principal=oryx/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:45.716820 3619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:45.717025 3619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:45.723572 3627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:45.723580 3629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:45.723817 3626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:45.723861 3619 server_base.cc:1034] running on GCE node
I20260807 08:42:45.724327 3619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:45.724682 3619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:45.725895 3619 hybrid_clock.cc:648] HybridClock initialized: now 1786092165725860 us; error 57 us; skew 500 ppm
Aug 07 08:42:45 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:45.729171 3619 init.cc:377] Logged in from keytab as oryx/127.28.108.131@KRBTEST.COM (short username oryx)
I20260807 08:42:45.729846 3619 webserver.cc:460] Webserver started at http://127.28.108.131:42925/ using document root <none> and password file <none>
I20260807 08:42:45.730140 3619 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:45.730216 3619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:45.730391 3619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:45.731573 3619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "7f78c58afe6a423a990b1d498b6a7939"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "15e019ce7e2ec749e949d1986d0098b9"
server_key_iv: "98f5e5d4d59a7dd877278f30dd0c24d2"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.731990 3619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "7f78c58afe6a423a990b1d498b6a7939"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "15e019ce7e2ec749e949d1986d0098b9"
server_key_iv: "98f5e5d4d59a7dd877278f30dd0c24d2"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.733940 3619 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 08:42:45.735067 3637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:45.735412 3619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:45.735658 3619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "7f78c58afe6a423a990b1d498b6a7939"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "15e019ce7e2ec749e949d1986d0098b9"
server_key_iv: "98f5e5d4d59a7dd877278f30dd0c24d2"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.735783 3619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:45.756979 3619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:45.759900 3619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:45.760113 3619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:45.760401 3619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:45.760797 3619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:45.760877 3619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:45.760921 3619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:45.761013 3619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:45.769125 3619 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:33891
I20260807 08:42:45.769201 3760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:33891 every 8 connection(s)
I20260807 08:42:45.769845 3619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
Aug 07 08:42:45 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:45.779922 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 3619
I20260807 08:42:45.780069 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:45.780316 29106 external_mini_cluster.cc:1428] Setting key 3fca33e45404ed63c363fbb2472ab293
I20260807 08:42:45.781605 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:45.770856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:58845 (local address 127.28.108.190:33641)
0807 08:42:45.770975 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:45.770978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:45.771163 (+ 185us) server_negotiation.cc:407] Connection header received
0807 08:42:45.771783 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:45.771785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:45.771798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:45.771845 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:45.773018 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.773545 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.774679 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.774908 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.775004 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:45.777948 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:45.777950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:45.777950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:45.777959 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:45.779530 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:45.780082 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:45.780084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:45.780085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:45.780165 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:45.780639 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:45.780640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:45.780641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:45.780853 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:45.780950 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:45.781386 (+ 436us) server_negotiation.cc:299] Negotiation successful
0807 08:42:45.781500 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 08:42:45.781919 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:45.770734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.131:58845)
0807 08:42:45.771000 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:45.771055 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:45.771642 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:45.772013 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:45.772017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:45.772663 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:45.772889 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:45.772895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.773679 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.773681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:45.774575 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:45.774577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:45.775018 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:45.775019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:45.775159 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:45.775767 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:45.775777 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:45.777741 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:45.779652 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:45.779664 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:45.779667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:45.779967 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:45.780252 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:45.780253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:45.780254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:45.780526 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:45.780962 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:45.780973 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:45.781205 (+ 232us) client_negotiation.cc:769] Sending connection context
0807 08:42:45.781427 (+ 222us) client_negotiation.cc:240] Negotiation successful
0807 08:42:45.781674 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":136,"threads_started":1}
I20260807 08:42:45.782672 3761 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33641
I20260807 08:42:45.782804 3761 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:45.783025 3761 heartbeater.cc:511] Master 127.28.108.190:33641 requested a full tablet report, sending...
I20260807 08:42:45.783520 3260 ts_manager.cc:194] Registered new tserver with Master: 7f78c58afe6a423a990b1d498b6a7939 (127.28.108.131:33891)
I20260807 08:42:45.783927 3260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:58845
I20260807 08:42:45.792711 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.108.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.189.keytab.
Entry for principal oryx/127.28.108.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.189.keytab.
I20260807 08:42:45.826749 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.189:39863
--webserver_interface=127.28.108.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.189.keytab
--principal=oryx/127.28.108.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:33641,127.28.108.189:39863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:45.934830 3767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:45.935124 3767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:45.935174 3767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:45.936589 3767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:45.936682 3767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:45.936712 3767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:45.936746 3767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:45.936781 3767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:45.938720 3767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:33641,127.28.108.189:39863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.189.keytab
--principal=oryx/127.28.108.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.189:39863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.108.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
W20260807 08:42:45.938895 3767 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.108.190:33641,127.28.108.189:39863'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 08:42:45.939092 3767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:45.939389 3767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:45.945689 3775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:45.945731 3777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:45.945689 3774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:45.946158 3767 server_base.cc:1034] running on GCE node
I20260807 08:42:45.946435 3767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:45.946770 3767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:45.947964 3767 hybrid_clock.cc:648] HybridClock initialized: now 1786092165947914 us; error 71 us; skew 500 ppm
Aug 07 08:42:45 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:45.950989 3767 init.cc:377] Logged in from keytab as oryx/127.28.108.189@KRBTEST.COM (short username oryx)
I20260807 08:42:45.951439 3767 webserver.cc:460] Webserver started at http://127.28.108.189:43217/ using document root <none> and password file <none>
I20260807 08:42:45.951723 3767 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:45.951807 3767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:45.951969 3767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:45.953099 3767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data/instance:
uuid: "71b672a0d5af4ae3b409073cf266f386"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "8412e5689c163d07e1a8153dbd4847d4"
server_key_iv: "a15201f64653cff455c0189a70a28925"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.953563 3767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/wal/instance:
uuid: "71b672a0d5af4ae3b409073cf266f386"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "8412e5689c163d07e1a8153dbd4847d4"
server_key_iv: "a15201f64653cff455c0189a70a28925"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.955325 3767 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:42:45.956593 3784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:45.956838 3767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:42:45.956954 3767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/wal
uuid: "71b672a0d5af4ae3b409073cf266f386"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "8412e5689c163d07e1a8153dbd4847d4"
server_key_iv: "a15201f64653cff455c0189a70a28925"
server_key_version: "encryptionkey@0"
I20260807 08:42:45.957084 3767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:45.996564 3767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:45.999351 3767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:45.999492 3767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:46.004267 3767 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.189:39863
I20260807 08:42:46.004458 3837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.189:39863 every 8 connection(s)
I20260807 08:42:46.004791 3767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
I20260807 08:42:46.006060 3838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:46.007845 3838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.189@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:46.014736 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 3767
I20260807 08:42:46.019639 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.008748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.28.108.190:33641)
0807 08:42:46.008825 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.008827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.009151 (+ 324us) server_negotiation.cc:407] Connection header received
0807 08:42:46.009650 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.009651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.009665 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.009728 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.010565 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.011081 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.012265 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.012470 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.012534 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.015457 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.015459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.015460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.015471 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.017576 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.018300 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.018302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.018302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.018393 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.018727 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.018728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.018728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.018956 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.019040 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.019409 (+ 369us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.019527 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 08:42:46.019897 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.008695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:48802)
0807 08:42:46.008946 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.008987 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.009500 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.009894 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.009898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.010182 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.010452 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.010457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.011178 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.011178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.012081 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.012084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.012564 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.012565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.012681 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.013466 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.013476 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.015279 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.017740 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.017748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.017750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.018127 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.018498 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.018499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.018499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.018635 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.019063 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.019070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.019225 (+ 155us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.019437 (+ 212us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.019632 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":143,"threads_started":1}
I20260807 08:42:46.020792 3838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.108.189@KRBTEST.COM
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.189@KRBTEST.COM for oryx/127.28.108.189@KRBTEST.COM
I20260807 08:42:46.026876 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.015546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.28.108.189:39863)
0807 08:42:46.015772 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.015778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.015808 (+ 30us) server_negotiation.cc:407] Connection header received
0807 08:42:46.015836 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.015839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.015861 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.015932 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.016761 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.018083 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.019314 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.019718 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.019800 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.023119 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.023130 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.023131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.023139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.025296 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.025847 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.025859 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.025859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.025923 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.026178 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.026178 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.026179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.026337 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.026570 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.026621 (+ 51us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.026713 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":151,"threads_started":1}
I20260807 08:42:46.029994 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.021106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:39863 (local address 127.0.0.1:45592)
0807 08:42:46.021154 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.021164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.021204 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.021527 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.021528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.021713 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.021768 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.021770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.022779 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.022780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.023785 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.023788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.024212 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.024213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.024284 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.024763 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.024768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.026233 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.028720 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.028721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.028722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.028893 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.029103 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.029104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.029105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.029163 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.029605 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.029606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.029684 (+ 78us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.029759 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.029845 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260807 08:42:46.030256 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.021174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.28.108.189:39863)
0807 08:42:46.021401 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.021403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.021411 (+ 8us) server_negotiation.cc:407] Connection header received
0807 08:42:46.021431 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.021431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.021442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.021525 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.021877 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.022671 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.023886 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.024125 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.024214 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.026330 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.026331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.026332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.026339 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.028633 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.028965 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.028966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.028967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.029028 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.029327 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.029328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.029329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.029464 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.029558 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.029829 (+ 271us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.029920 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":149,"threads_started":1}
I20260807 08:42:46.032238 3838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: Bootstrap starting.
I20260807 08:42:46.033001 3838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:46.033392 3838 log.cc:824] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:46.034301 3838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: No bootstrap required, opened a new log
I20260807 08:42:46.035943 3838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } }
I20260807 08:42:46.036098 3838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:46.036124 3838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b672a0d5af4ae3b409073cf266f386, State: Initialized, Role: FOLLOWER
I20260807 08:42:46.036283 3838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } }
I20260807 08:42:46.036603 3838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:46.036629 3845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } } }
I20260807 08:42:46.036852 3845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 08:42:46.038892 3856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:42:46.038970 3856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:42:46.041723 3767 master_runner.cc:428] Detected that this master 71b672a0d5af4ae3b409073cf266f386 is joining an existing cluster
I20260807 08:42:46.041846 3767 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.108.189:39863
I20260807 08:42:46.051558 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.043578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.28.108.189:39863)
0807 08:42:46.043629 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.043632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.043643 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:42:46.043702 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.043703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.043714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.043788 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.044374 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.045261 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.046387 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.046637 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.046746 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.048014 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.048016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.048016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.048024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.050134 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.050635 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.050637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.050637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.050699 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.050961 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.050962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.050963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.051116 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.051162 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.051365 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.051461 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 08:42:46.051801 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.043398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:48804)
0807 08:42:46.043745 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.043754 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.043858 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.044058 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.044059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.044262 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.044322 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.044323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.045102 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.045103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.046201 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.046203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.046813 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.046814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.046909 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.047601 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.047607 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.048112 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.050411 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.050412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.050413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.050727 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.050996 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.050997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.050997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.051083 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.051393 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.051395 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.051444 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.051538 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.051668 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":98,"threads_started":1}
I20260807 08:42:46.052143 3863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.043380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:39863 (local address 127.0.0.1:45594)
0807 08:42:46.043559 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.043573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.043636 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.043812 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.043813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.044156 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.044273 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.044276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.045356 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.045357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.046315 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.046318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.046746 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.046747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.046836 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.047401 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.047407 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.047891 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.050240 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.050241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.050242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.050547 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.050786 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.050787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.050787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.050873 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.051211 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.051212 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.051254 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.051794 (+ 540us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.052025 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260807 08:42:46.052168 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.043421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.28.108.190:33641)
0807 08:42:46.043480 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.043483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.043828 (+ 345us) server_negotiation.cc:407] Connection header received
0807 08:42:46.043946 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.043947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.043962 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.044034 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.044433 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.045001 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.046308 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.046665 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.046746 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.048221 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.048222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.048223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.048232 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.050294 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.050845 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.050846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.050847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.050921 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.051158 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.051159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.051160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.051304 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.051389 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.051573 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.052095 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.190@KRBTEST.COM for oryx/127.28.108.189@KRBTEST.COM
I20260807 08:42:46.064685 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.055395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:39863 (local address 127.0.0.1:45606)
0807 08:42:46.055563 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.055612 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.055711 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.055907 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.055909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.056310 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.056427 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.056430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.057488 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.057489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.058503 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.058507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.059054 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.059055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.059126 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.059766 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.059772 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.061293 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.063388 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.063397 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.063399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.063650 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.063905 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.063906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.063906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.063980 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.064306 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.064316 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.064380 (+ 64us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.064460 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.064585 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":112,"threads_started":1}
I20260807 08:42:46.064719 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.055473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.28.108.189:39863)
0807 08:42:46.055519 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.055522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.055672 (+ 150us) server_negotiation.cc:407] Connection header received
0807 08:42:46.055810 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.055811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.055823 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.055903 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.056515 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.057376 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.058593 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.058961 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.059049 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.061414 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.061416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.061417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.061427 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.063269 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.063737 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.063739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.063739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.063813 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.064059 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.064060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.064061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.064224 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.064305 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.064514 (+ 209us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.064610 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 08:42:46.065951 3260 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.108.189:39863: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "71b672a0d5af4ae3b409073cf266f386"
member_type: NON_VOTER
last_known_addr {
host: "127.28.108.189"
port: 39863
}
attrs {
promote: true
}
}
dest_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0"
cas_config_opid_index: -1
I20260807 08:42:46.066479 3260 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } }
I20260807 08:42:46.067057 3867 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 71b672a0d5af4ae3b409073cf266f386 (127.28.108.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } } }
I20260807 08:42:46.067483 3812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:46.068445 3867 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.108.189:39863
I20260807 08:42:46.068554 3867 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } } }
I20260807 08:42:46.068650 3867 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:46.068807 3812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Refusing update from remote peer ed8bac22a00c4500bdc8eeebc76d77a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260807 08:42:46.069206 3767 master.cc:537] Master@127.28.108.189:39863 shutting down...
I20260807 08:42:46.070087 3866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 08:42:46.071595 3246 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260807 08:42:46.072096 3246 consensus_peers.cc:487] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 -> Peer 71b672a0d5af4ae3b409073cf266f386 (127.28.108.189:39863): Couldn't send request to peer 71b672a0d5af4ae3b409073cf266f386. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:42:46.072546 3767 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:46.072651 3767 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:46.072695 3767 tablet_replica.cc:330] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: stopping tablet replica
I20260807 08:42:46.086506 3767 master.cc:552] Master@127.28.108.189:39863 shutdown complete.
I20260807 08:42:46.086570 3767 master_runner.cc:305] Clearing existing system tablet
I20260807 08:42:46.086733 3767 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:46.086787 3767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:46.087690 3767 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:42:46.088428 3871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.088613 3767 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:46.088677 3767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:46.088724 3767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/wal
uuid: "71b672a0d5af4ae3b409073cf266f386"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "8412e5689c163d07e1a8153dbd4847d4"
server_key_iv: "a15201f64653cff455c0189a70a28925"
server_key_version: "encryptionkey@0"
I20260807 08:42:46.089558 3767 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:46.091816 3767 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 08:42:46.091912 3767 log.cc:1196] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260807 08:42:46.092176 3767 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: Deleting consensus metadata
I20260807 08:42:46.092845 3767 master_runner.cc:315] Copying system tablet from 127.28.108.190:33641
I20260807 08:42:46.093798 3767 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: tablet copy: Beginning tablet copy session from remote peer at address 127.28.108.190:33641
I20260807 08:42:46.101990 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.094385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.28.108.190:33641)
0807 08:42:46.094443 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.094446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.094576 (+ 130us) server_negotiation.cc:407] Connection header received
0807 08:42:46.094740 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.094741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.094821 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.094905 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.095383 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.095969 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.096846 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.097076 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.097165 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.098498 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.098500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.098500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.098509 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.100290 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.100852 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.100853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.100854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.100934 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.101175 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.101176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.101176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.101433 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.101533 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.101761 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.101872 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 08:42:46.102020 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.094282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:48818)
0807 08:42:46.094470 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.094483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.094599 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.094950 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.094952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.095180 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.095275 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.095277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.096082 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.096083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.096723 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.096725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.097164 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.097165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.097259 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.097892 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.097897 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.098377 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.100513 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.100514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.100515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.100755 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.101020 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.101021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.101021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.101095 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.101536 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.101537 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.101604 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.101728 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.101889 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
I20260807 08:42:46.102767 3281 tablet_copy_service.cc:140] P ed8bac22a00c4500bdc8eeebc76d77a0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 71b672a0d5af4ae3b409073cf266f386 ({username='oryx', principal='oryx/127.28.108.189@KRBTEST.COM'} at 127.0.0.1:48818)
I20260807 08:42:46.102970 3281 tablet_copy_service.cc:161] P ed8bac22a00c4500bdc8eeebc76d77a0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 71b672a0d5af4ae3b409073cf266f386 at {username='oryx', principal='oryx/127.28.108.189@KRBTEST.COM'} at 127.0.0.1:48818: session id = 71b672a0d5af4ae3b409073cf266f386-00000000000000000000000000000000
I20260807 08:42:46.104068 3281 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:42:46.105052 3767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 08:42:46.106546 3767 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: tablet copy: Starting download of 0 data blocks...
I20260807 08:42:46.106740 3767 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: tablet copy: Starting download of 1 WAL segments...
I20260807 08:42:46.107985 3767 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:42:46.109046 3281 tablet_copy_service.cc:342] P ed8bac22a00c4500bdc8eeebc76d77a0: Request end of tablet copy session 71b672a0d5af4ae3b409073cf266f386-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.108.189@KRBTEST.COM'} at 127.0.0.1:48818
I20260807 08:42:46.109123 3281 tablet_copy_service.cc:433] P ed8bac22a00c4500bdc8eeebc76d77a0: ending tablet copy session 71b672a0d5af4ae3b409073cf266f386-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 71b672a0d5af4ae3b409073cf266f386
I20260807 08:42:46.112502 3767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:46.114352 3879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:46.114554 3878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:46.115402 3881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:46.115749 3767 server_base.cc:1034] running on GCE node
I20260807 08:42:46.115877 3767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:46.116039 3767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:46.117175 3767 hybrid_clock.cc:648] HybridClock initialized: now 1786092166117167 us; error 41 us; skew 500 ppm
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:46.119115 3767 init.cc:377] Logged in from keytab as oryx/127.28.108.189@KRBTEST.COM (short username oryx)
I20260807 08:42:46.119552 3767 webserver.cc:460] Webserver started at http://127.28.108.189:39641/ using document root <none> and password file <none>
I20260807 08:42:46.119714 3767 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:46.119767 3767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:46.120443 3767 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260807 08:42:46.121052 3888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.121230 3767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260807 08:42:46.121280 3767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/wal
uuid: "71b672a0d5af4ae3b409073cf266f386"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "8412e5689c163d07e1a8153dbd4847d4"
server_key_iv: "a15201f64653cff455c0189a70a28925"
server_key_version: "encryptionkey@0"
I20260807 08:42:46.121398 3767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:46.149081 3767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:46.151352 3767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:46.156086 3767 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.189:39863
I20260807 08:42:46.156104 3948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.189:39863 every 8 connection(s)
I20260807 08:42:46.156530 3767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
I20260807 08:42:46.157677 3949 sys_catalog.cc:263] Verifying existing consensus state
I20260807 08:42:46.158403 3949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: Bootstrap starting.
I20260807 08:42:46.162050 3949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:42:46.162384 3949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: Bootstrap complete.
I20260807 08:42:46.162624 3949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } }
I20260807 08:42:46.162730 3949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 71b672a0d5af4ae3b409073cf266f386, State: Initialized, Role: LEARNER
I20260807 08:42:46.162772 3949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } }
I20260807 08:42:46.163033 3949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:46.163017 3952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } } }
I20260807 08:42:46.163172 3952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: This master's current role is: LEARNER
I20260807 08:42:46.164947 3963 catalog_manager.cc:1259] Loaded cluster ID: 172eadb7b96140c092e0d4aede3b8c81
I20260807 08:42:46.165009 3963 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: loading cluster ID for follower catalog manager: success
I20260807 08:42:46.166457 3963 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: acquiring CA information for follower catalog manager: success
I20260807 08:42:46.166816 3963 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 08:42:46.188261 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.179634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.28.108.189:39863)
0807 08:42:46.179820 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.179822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.179830 (+ 8us) server_negotiation.cc:407] Connection header received
0807 08:42:46.179857 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.179858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.179869 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.179963 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.180665 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.181466 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.182404 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.182654 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.182748 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.184420 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.184421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.184422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.184428 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.186392 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.186943 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.186944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.186945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.187016 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.187359 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.187360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.187361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.187497 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.187665 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.187892 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.188003 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for oryx/127.28.108.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.188.keytab.
Entry for principal oryx/127.28.108.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.188.keytab.
I20260807 08:42:46.225978 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.188:37301
--webserver_interface=127.28.108.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.188.keytab
--principal=oryx/127.28.108.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:33641,127.28.108.189:39863,127.28.108.188:37301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:46.324018 3967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:46.324254 3967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:46.324299 3967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:46.325714 3967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:46.325771 3967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:46.325786 3967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:46.325799 3967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:46.325811 3967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:46.327476 3967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:33641,127.28.108.189:39863,127.28.108.188:37301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.188.keytab
--principal=oryx/127.28.108.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.188:37301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.108.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:46.327785 3967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:46.328045 3967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:42:46.333801 3967 server_base.cc:1034] running on GCE node
W20260807 08:42:46.333740 3973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:46.333720 3975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:46.333717 3972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:46.334321 3967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:46.334684 3967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:46.335863 3967 hybrid_clock.cc:648] HybridClock initialized: now 1786092166335827 us; error 58 us; skew 500 ppm
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:46.339037 3967 init.cc:377] Logged in from keytab as oryx/127.28.108.188@KRBTEST.COM (short username oryx)
I20260807 08:42:46.339596 3967 webserver.cc:460] Webserver started at http://127.28.108.188:46167/ using document root <none> and password file <none>
I20260807 08:42:46.339929 3967 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:46.340014 3967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:46.340191 3967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:46.341498 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data/instance:
uuid: "11263b30c20a4565833be137b6e3bd5c"
format_stamp: "Formatted at 2026-08-07 08:42:46 on dist-test-slave-9zdj"
server_key: "d023e2ed9c42c34bc1558d9c2d86723f"
server_key_iv: "8ea8d9fa60bf37b8140b8ec3eab884ab"
server_key_version: "encryptionkey@0"
I20260807 08:42:46.341998 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/wal/instance:
uuid: "11263b30c20a4565833be137b6e3bd5c"
format_stamp: "Formatted at 2026-08-07 08:42:46 on dist-test-slave-9zdj"
server_key: "d023e2ed9c42c34bc1558d9c2d86723f"
server_key_iv: "8ea8d9fa60bf37b8140b8ec3eab884ab"
server_key_version: "encryptionkey@0"
I20260807 08:42:46.343604 3967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:42:46.344580 3982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.344774 3967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:46.344877 3967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/wal
uuid: "11263b30c20a4565833be137b6e3bd5c"
format_stamp: "Formatted at 2026-08-07 08:42:46 on dist-test-slave-9zdj"
server_key: "d023e2ed9c42c34bc1558d9c2d86723f"
server_key_iv: "8ea8d9fa60bf37b8140b8ec3eab884ab"
server_key_version: "encryptionkey@0"
I20260807 08:42:46.344964 3967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:46.369426 3967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:46.372205 3967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:46.372347 3967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:46.376716 3967 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.188:37301
I20260807 08:42:46.376758 4038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.188:37301 every 8 connection(s)
I20260807 08:42:46.376991 3465 heartbeater.cc:503] Master 127.28.108.190:33641 was elected leader, sending a full tablet report...
I20260807 08:42:46.377230 3967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
I20260807 08:42:46.378464 4039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:46.380128 4039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:42:46.383702 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 3967
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.188@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.108.188@KRBTEST.COM
I20260807 08:42:46.392181 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.381100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.28.108.190:33641)
0807 08:42:46.381167 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.381169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.381183 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:42:46.381801 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.381803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.381816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.381878 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.382583 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.383119 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.384798 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.385147 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.385236 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.387876 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.387878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.387878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.387888 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.389945 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.390461 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.390462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.390463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.390575 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.390951 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.390952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.390953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.391139 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.391521 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.391933 (+ 412us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.392046 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 08:42:46.392392 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.380839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:48822)
0807 08:42:46.381031 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.381077 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.381545 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.381974 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.381978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.382269 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.382480 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.382484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.383300 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.383301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.384705 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.384707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.385419 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.385420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.385606 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.386142 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.386152 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.387776 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.390046 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.390053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.390055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.390360 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.390670 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.390672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.390672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.390838 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.391550 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.391558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.391753 (+ 195us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.391885 (+ 132us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.392088 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260807 08:42:46.393180 4039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:42:46.394306 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.384357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.28.108.188:37301)
0807 08:42:46.384541 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.384545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.384563 (+ 18us) server_negotiation.cc:407] Connection header received
0807 08:42:46.384586 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.384588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.384608 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.384697 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.385495 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.386537 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.387307 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.387704 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.387785 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.390957 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.390965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.390966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.390973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.392856 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.393258 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.393262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.393262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.393354 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.393600 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.393601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.393601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.393842 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.393979 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.394097 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.394207 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.188@KRBTEST.COM for oryx/127.28.108.189@KRBTEST.COM
I20260807 08:42:46.400698 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.393573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.28.108.189:39863)
0807 08:42:46.393676 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.393678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.393709 (+ 31us) server_negotiation.cc:407] Connection header received
0807 08:42:46.393763 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.393764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.393775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.393843 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.394178 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.394969 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.395657 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.395955 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.396038 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.398115 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.398117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.398117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.398127 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.399532 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.399885 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.399886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.399887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.399981 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.400221 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.400222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.400222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.400347 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.400404 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.400552 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.400629 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 08:42:46.400733 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.393501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:39863 (local address 127.0.0.1:45656)
0807 08:42:46.393659 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.393670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.393718 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.393844 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.393844 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.394026 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.394094 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.394095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.395057 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.395058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.395556 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.395558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.396038 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.396039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.396099 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.396547 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.396552 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.398023 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.399623 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.399624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.399624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.399797 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.400078 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.400079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.400079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.400145 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.400412 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.400413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.400459 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.400518 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.400611 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260807 08:42:46.401409 4039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.188@KRBTEST.COM for oryx/127.28.108.188@KRBTEST.COM
I20260807 08:42:46.408111 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.401739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:37301 (local address 127.0.0.1:46098)
0807 08:42:46.401818 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.401831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.401871 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.402002 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.402003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.402151 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.402185 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.402187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.402959 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.402959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.403440 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.403441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.403833 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.403834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.403882 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.404334 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.404338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.405736 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.407127 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.407128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.407129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.407326 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.407579 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.407580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.407580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.407640 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.407858 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.407859 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.407904 (+ 45us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.407954 (+ 50us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.408040 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 08:42:46.408253 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.401796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.28.108.188:37301)
0807 08:42:46.401851 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.401852 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.401875 (+ 23us) server_negotiation.cc:407] Connection header received
0807 08:42:46.401932 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.401933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.401944 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.401995 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.402377 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.402873 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.403536 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.403759 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.403839 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.405818 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.405820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.405820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.405827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.407015 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.407435 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.407436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.407436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.407498 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.407684 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.407684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.407684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.407800 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.407852 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.407983 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.408101 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":20}
I20260807 08:42:46.410315 4039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: Bootstrap starting.
I20260807 08:42:46.411223 4039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:46.411541 4039 log.cc:824] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:46.412377 4039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: No bootstrap required, opened a new log
I20260807 08:42:46.414055 4039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } }
I20260807 08:42:46.414196 4039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:46.414219 4039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11263b30c20a4565833be137b6e3bd5c, State: Initialized, Role: FOLLOWER
I20260807 08:42:46.414386 4039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } }
I20260807 08:42:46.414733 4039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:46.414711 4045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } } }
I20260807 08:42:46.414866 4045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: This master's current role is: FOLLOWER
W20260807 08:42:46.416668 4056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:42:46.416760 4056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:42:46.419948 3967 master_runner.cc:428] Detected that this master 11263b30c20a4565833be137b6e3bd5c is joining an existing cluster
I20260807 08:42:46.420024 3967 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.108.188:37301
I20260807 08:42:46.429247 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.421649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:48834)
0807 08:42:46.422009 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.422018 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.422095 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.422322 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.422323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.422607 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.422744 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.422747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.423542 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.423543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.424558 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.424561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.425078 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.425079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.425144 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.425694 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.425700 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.426122 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.428075 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.428076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.428077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.428276 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.428506 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.428507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.428507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.428567 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.428868 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.428869 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.428926 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.428985 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.429075 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260807 08:42:46.429270 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.421975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.28.108.190:33641)
0807 08:42:46.422027 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.422029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.422069 (+ 40us) server_negotiation.cc:407] Connection header received
0807 08:42:46.422174 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.422176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.422188 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.422221 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.422847 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.423411 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.424662 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.424984 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.425066 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.426243 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.426244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.426245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.426253 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.427953 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.428363 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.428364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.428365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.428434 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.428642 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.428642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.428643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.428794 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.428869 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.429043 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.429164 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092165, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.190@KRBTEST.COM for oryx/127.28.108.188@KRBTEST.COM
I20260807 08:42:46.445978 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 3325
I20260807 08:42:46.452459 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.431716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:37301 (local address 127.0.0.1:46120)
0807 08:42:46.431802 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.431815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.431866 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.432390 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.432391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.432581 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.432634 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.432637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.433499 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.433500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.434385 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.434388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.434764 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.434766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.434853 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.435431 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.435437 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.436978 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.450534 (+ 13556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.450536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.450536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.450814 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.451611 (+ 797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.451613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.451614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.451712 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.452043 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.452044 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.452101 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.452153 (+ 52us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.452299 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260807 08:42:46.453578 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.431791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.28.108.188:37301)
0807 08:42:46.432219 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.432221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.432229 (+ 8us) server_negotiation.cc:407] Connection header received
0807 08:42:46.432264 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.432265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.432292 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.432379 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.432729 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.433323 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.434475 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.434685 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.434748 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.437106 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.437113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.437114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.437121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.449567 (+ 12446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.450912 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.450913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.450913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.450981 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.451796 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.451796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.451797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.451964 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.452674 (+ 710us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.452723 (+ 49us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.452826 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":241,"threads_started":1}
I20260807 08:42:46.454559 3260 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.108.188:37301: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "11263b30c20a4565833be137b6e3bd5c"
member_type: NON_VOTER
last_known_addr {
host: "127.28.108.188"
port: 37301
}
attrs {
promote: true
}
}
dest_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0"
cas_config_opid_index: 5
I20260807 08:42:46.454879 3260 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } }
I20260807 08:42:46.455380 3867 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 11263b30c20a4565833be137b6e3bd5c (127.28.108.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.456022 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:46.456485 3867 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.108.188:37301
I20260807 08:42:46.456570 3867 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.456636 3867 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:46.456969 3967 master.cc:537] Master@127.28.108.188:37301 shutting down...
I20260807 08:42:46.457113 4011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 FOLLOWER]: Refusing update from remote peer ed8bac22a00c4500bdc8eeebc76d77a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 08:42:46.457506 3866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:46.458703 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.422253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.28.108.188:37301)
0807 08:42:46.422293 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.422296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.422302 (+ 6us) server_negotiation.cc:407] Connection header received
0807 08:42:46.422320 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.422321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.422330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.422380 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.446348 (+ 23968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.447215 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.449708 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.450118 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.450176 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.452858 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.452859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.452859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.452867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.456767 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.457589 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.457590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.457591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.457686 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.458010 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.458011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.458012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.458162 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.458214 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.458462 (+ 248us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.458563 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260807 08:42:46.459103 3246 consensus_peers.cc:487] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 -> Peer 11263b30c20a4565833be137b6e3bd5c (127.28.108.188:37301): Couldn't send request to peer 11263b30c20a4565833be137b6e3bd5c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:42:46.459422 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.421968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:37301 (local address 127.0.0.1:46112)
0807 08:42:46.422083 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.422094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.422147 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.422424 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.422425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.446136 (+ 23711us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.446240 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.446244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.447355 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.447356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.448306 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.448310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.450268 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.450268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.450333 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.451060 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.451064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.451485 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.457236 (+ 5751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.457237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.457238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.457499 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.457858 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.457859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.457859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.457943 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.458265 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.458266 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.458327 (+ 61us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.459025 (+ 698us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.459240 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260807 08:42:46.459795 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:37387
--local_ip_for_outbound_sockets=127.28.108.129
--tserver_master_addrs=127.28.108.190:33641
--webserver_port=42139
--webserver_interface=127.28.108.129
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab
--principal=oryx/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.108.190:33641,127.28.108.189:39863
--tserver_master_addrs=127.28.108.190:33641,127.28.108.189:39863,127.28.108.188:37301 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:42:46.461042 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.422208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.28.108.189:39863)
0807 08:42:46.422256 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.422258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.422267 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:42:46.422295 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.422297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.422316 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.422383 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.422869 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.423661 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.425155 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.425474 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.425543 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.452497 (+ 26954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.452500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.452501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.452511 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.454573 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.455003 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.455004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.455004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.455074 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.460349 (+ 5275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.460352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.460352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.460541 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.460598 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.460835 (+ 237us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.460946 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 08:42:46.461671 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.421648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:39863 (local address 127.0.0.1:45658)
0807 08:42:46.421835 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.421848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.421911 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.422455 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.422456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.422695 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.422769 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.422771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.423747 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.423748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.424558 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.424561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.425602 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.425603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.425718 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.448433 (+ 22715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.448440 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.449157 (+ 717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.454683 (+ 5526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.454684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.454684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.454911 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.455149 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.455150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.455150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.455322 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.460658 (+ 5336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.460658 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.460715 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.461387 (+ 672us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.461517 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":4,"thread_start_us":105,"threads_started":1}
I20260807 08:42:46.464075 3967 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:46.464215 3967 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:46.464273 3967 tablet_replica.cc:330] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: stopping tablet replica
I20260807 08:42:46.464408 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.455542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:39863 (local address 127.0.0.1:45672)
0807 08:42:46.457019 (+ 1477us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.457032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.457074 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.459396 (+ 2322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.459398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.459400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:46.459463 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.459466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.460723 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.460725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.462109 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.462114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.464093 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.464094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.464189 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.464232 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.464272 (+ 40us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.464328 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1452}
I20260807 08:42:46.465083 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.456255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.28.108.189:39863)
0807 08:42:46.459173 (+ 2918us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.459175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.459185 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:46.459214 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.459215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.459247 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.459545 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:46.459595 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.460623 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.462462 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.463976 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.464763 (+ 787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.464858 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.464894 (+ 36us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.464983 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2891,"thread_start_us":76,"threads_started":1}
I20260807 08:42:46.465534 3920 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 LEARNER]: Refusing update from remote peer ed8bac22a00c4500bdc8eeebc76d77a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 08:42:46.465929 3866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 08:42:46.466707 3920 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 11263b30c20a4565833be137b6e3bd5c (127.28.108.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.467955 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.467791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:37301 (local address 127.0.0.1:46128)
0807 08:42:46.467872 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.467912 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260807 08:42:46.468382 3952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed8bac22a00c4500bdc8eeebc76d77a0. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.468497 3952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: This master's current role is: LEARNER
I20260807 08:42:46.468429 4070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: NON_VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: true } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.468538 4070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: This master's current role is: LEARNER
I20260807 08:42:46.468649 3866 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0: attempting to promote NON_VOTER 71b672a0d5af4ae3b409073cf266f386 to VOTER
I20260807 08:42:46.468945 3866 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } }
I20260807 08:42:46.469753 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.469457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:37301 (local address 127.0.0.1:46142)
0807 08:42:46.469527 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.469605 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 08:42:46.469841 3920 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 LEARNER]: Refusing update from remote peer ed8bac22a00c4500bdc8eeebc76d77a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 08:42:46.470100 3867 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 08:42:46.470448 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.470327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:37301 (local address 127.0.0.1:46156)
0807 08:42:46.470375 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.470411 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260807 08:42:46.470728 3246 consensus_peers.cc:487] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 -> Peer 11263b30c20a4565833be137b6e3bd5c (127.28.108.188:37301): Couldn't send request to peer 11263b30c20a4565833be137b6e3bd5c. Status: Network error: Client connection negotiation failed: client connection to 127.28.108.188:37301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 08:42:46.471108 4071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 71b672a0d5af4ae3b409073cf266f386 (127.28.108.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.471227 3920 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 71b672a0d5af4ae3b409073cf266f386 (127.28.108.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.472594 4070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.472675 3867 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.472782 3867 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:46.472759 4070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:42:46.472903 3867 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.473006 3867 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:46.478296 3967 master.cc:552] Master@127.28.108.188:37301 shutdown complete.
I20260807 08:42:46.478381 3967 master_runner.cc:305] Clearing existing system tablet
I20260807 08:42:46.478535 3967 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:46.478590 3967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:46.479466 3967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:42:46.480086 4074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.480275 3967 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:46.480340 3967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:46.480370 3967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/wal
uuid: "11263b30c20a4565833be137b6e3bd5c"
format_stamp: "Formatted at 2026-08-07 08:42:46 on dist-test-slave-9zdj"
server_key: "d023e2ed9c42c34bc1558d9c2d86723f"
server_key_iv: "8ea8d9fa60bf37b8140b8ec3eab884ab"
server_key_version: "encryptionkey@0"
I20260807 08:42:46.481271 3967 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:46.483188 3967 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 08:42:46.483289 3967 log.cc:1196] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 08:42:46.483541 3967 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: Deleting consensus metadata
I20260807 08:42:46.483673 3967 master_runner.cc:315] Copying system tablet from 127.28.108.190:33641
I20260807 08:42:46.484354 3967 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.108.190:33641
I20260807 08:42:46.492923 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.484837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.28.108.190:33641)
0807 08:42:46.484898 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.484900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.484974 (+ 74us) server_negotiation.cc:407] Connection header received
0807 08:42:46.485068 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.485069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.485081 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.485187 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.485863 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.486692 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.487939 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.488263 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.488338 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.489671 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.489673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.489673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.489681 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.491511 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.491981 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.491982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.491983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.492056 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.492304 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.492305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.492305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.492461 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.492532 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.492734 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.492825 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 08:42:46.492974 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.484748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:48842)
0807 08:42:46.484906 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.484919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.484991 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.485215 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.485217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.485461 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.485538 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.485541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.486800 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.486801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.487838 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.487842 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.488359 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.488360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.488444 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.489047 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.489053 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.489537 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.491604 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.491605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.491606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.491891 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.492137 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.492137 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.492138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.492231 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.492538 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.492539 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.492601 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.492694 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.492834 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260807 08:42:46.493841 3281 tablet_copy_service.cc:140] P ed8bac22a00c4500bdc8eeebc76d77a0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 11263b30c20a4565833be137b6e3bd5c ({username='oryx', principal='oryx/127.28.108.188@KRBTEST.COM'} at 127.0.0.1:48842)
I20260807 08:42:46.493918 3281 tablet_copy_service.cc:161] P ed8bac22a00c4500bdc8eeebc76d77a0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 11263b30c20a4565833be137b6e3bd5c at {username='oryx', principal='oryx/127.28.108.188@KRBTEST.COM'} at 127.0.0.1:48842: session id = 11263b30c20a4565833be137b6e3bd5c-00000000000000000000000000000000
I20260807 08:42:46.494593 3281 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:42:46.495306 3967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 08:42:46.496996 3967 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: tablet copy: Starting download of 0 data blocks...
I20260807 08:42:46.497299 3967 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: tablet copy: Starting download of 1 WAL segments...
I20260807 08:42:46.498915 3967 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:42:46.500166 3281 tablet_copy_service.cc:342] P ed8bac22a00c4500bdc8eeebc76d77a0: Request end of tablet copy session 11263b30c20a4565833be137b6e3bd5c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.108.188@KRBTEST.COM'} at 127.0.0.1:48842
I20260807 08:42:46.500241 3281 tablet_copy_service.cc:433] P ed8bac22a00c4500bdc8eeebc76d77a0: ending tablet copy session 11263b30c20a4565833be137b6e3bd5c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 11263b30c20a4565833be137b6e3bd5c
I20260807 08:42:46.504004 3967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:46.505654 4082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:46.505877 3967 server_base.cc:1034] running on GCE node
W20260807 08:42:46.505856 4083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:46.510908 4085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:46.511135 3967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:46.511341 3967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:46.512504 3967 hybrid_clock.cc:648] HybridClock initialized: now 1786092166512475 us; error 45 us; skew 500 ppm
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:46.514389 3967 init.cc:377] Logged in from keytab as oryx/127.28.108.188@KRBTEST.COM (short username oryx)
I20260807 08:42:46.514721 3967 webserver.cc:460] Webserver started at http://127.28.108.188:45585/ using document root <none> and password file <none>
I20260807 08:42:46.514851 3967 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:46.514890 3967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:46.515749 3967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:46.516438 4093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.516638 3967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:46.516700 3967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/wal
uuid: "11263b30c20a4565833be137b6e3bd5c"
format_stamp: "Formatted at 2026-08-07 08:42:46 on dist-test-slave-9zdj"
server_key: "d023e2ed9c42c34bc1558d9c2d86723f"
server_key_iv: "8ea8d9fa60bf37b8140b8ec3eab884ab"
server_key_version: "encryptionkey@0"
I20260807 08:42:46.516767 3967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:46.529892 3967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:46.532052 3967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:46.535553 3967 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.188:37301
I20260807 08:42:46.535612 4147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.188:37301 every 8 connection(s)
I20260807 08:42:46.535940 3967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
I20260807 08:42:46.537359 4148 sys_catalog.cc:263] Verifying existing consensus state
I20260807 08:42:46.538069 4148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: Bootstrap starting.
I20260807 08:42:46.541392 4148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:42:46.541755 4148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: Bootstrap complete.
I20260807 08:42:46.541990 4148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } }
I20260807 08:42:46.542094 4148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 11263b30c20a4565833be137b6e3bd5c, State: Initialized, Role: LEARNER
I20260807 08:42:46.542132 4148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } }
I20260807 08:42:46.542359 4148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:46.544006 4162 catalog_manager.cc:1259] Loaded cluster ID: 172eadb7b96140c092e0d4aede3b8c81
I20260807 08:42:46.544157 4162 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: loading cluster ID for follower catalog manager: success
I20260807 08:42:46.544883 4151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.545012 4151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: This master's current role is: LEARNER
I20260807 08:42:46.546021 4162 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: acquiring CA information for follower catalog manager: success
I20260807 08:42:46.546392 4162 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 08:42:46.576970 4067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:46.577267 4067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:46.577316 4067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:46.578703 4067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:46.578790 4067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:46.578840 4067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:46.581279 4067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab
--principal=oryx/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:37387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=42139
--tserver_master_addrs=127.28.108.190:33641,127.28.108.189:39863,127.28.108.188:37301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:46.581648 4067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:46.581943 4067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:46.587899 4167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:46.587932 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:46.587898 4168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:46.588263 4067 server_base.cc:1034] running on GCE node
I20260807 08:42:46.590039 4067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:46.590430 4067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:46.591639 4067 hybrid_clock.cc:648] HybridClock initialized: now 1786092166591566 us; error 92 us; skew 500 ppm
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:46.595861 4067 init.cc:377] Logged in from keytab as oryx/127.28.108.129@KRBTEST.COM (short username oryx)
I20260807 08:42:46.596558 4067 webserver.cc:460] Webserver started at http://127.28.108.129:42139/ using document root <none> and password file <none>
I20260807 08:42:46.596912 4067 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:46.597038 4067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:46.597153 3611 heartbeater.cc:503] Master 127.28.108.190:33641 was elected leader, sending a full tablet report...
I20260807 08:42:46.599289 4067 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:46.600204 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.600529 4067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:46.600679 4067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "ffee67b8ac02fca8315a37cb83f678ef"
server_key_iv: "57e53792a56b66a7348dc38d98760827"
server_key_version: "encryptionkey@0"
I20260807 08:42:46.601256 4067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:46.620025 4067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:46.623176 4067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:46.623399 4067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:46.623687 4067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:46.624063 4067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:46.624152 4067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.624224 4067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:46.624259 4067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.632305 4067 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:37387
I20260807 08:42:46.632390 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:37387 every 8 connection(s)
I20260807 08:42:46.632815 4067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:42:46.639945 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 4067
I20260807 08:42:46.640130 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 3471
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.189@KRBTEST.COM
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.188@KRBTEST.COM
I20260807 08:42:46.656675 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:41927
--local_ip_for_outbound_sockets=127.28.108.130
--tserver_master_addrs=127.28.108.190:33641
--webserver_port=37111
--webserver_interface=127.28.108.130
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab
--principal=oryx/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.108.190:33641,127.28.108.189:39863
--tserver_master_addrs=127.28.108.190:33641,127.28.108.189:39863,127.28.108.188:37301 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:42:46.657480 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.634417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:41851 (local address 127.28.108.189:39863)
0807 08:42:46.634463 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.634465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.634479 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:42:46.635413 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.635415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.635427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.635493 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.636289 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.636822 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.638345 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.638549 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.638637 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.652548 (+ 13911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.652550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.652551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.652561 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.655346 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.656063 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.656064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.656065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.656142 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.656544 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.656546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.656546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.656741 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.656800 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.657172 (+ 372us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.657313 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 08:42:46.657605 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.633796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:37301 (local address 127.28.108.129:34167)
0807 08:42:46.634242 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.634252 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.635210 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.635685 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.635686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.635997 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.636182 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.636188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.637388 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.637389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.638132 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.638135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.638718 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.638719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.638808 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.639587 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.639600 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.652379 (+ 12779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.655261 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.655280 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.655285 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.655947 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.656300 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.656301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.656301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.656445 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.656775 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.656785 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.656996 (+ 211us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.657123 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.657334 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":14,"spinlock_wait_cycles":8064,"thread_start_us":153,"threads_started":1}
I20260807 08:42:46.658007 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.634190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:39863 (local address 127.28.108.129:41851)
0807 08:42:46.634345 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.634355 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.635210 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.635513 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.635513 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.635958 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.636181 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.636186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.636960 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.636960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.638208 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.638211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.638661 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.638662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.638805 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.639668 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.639693 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.652350 (+ 12657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.655446 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.655447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.655448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.655948 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.656236 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.656237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.656238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.656440 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.656862 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.656863 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.656996 (+ 133us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.657577 (+ 581us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.657829 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260807 08:42:46.658133 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.633812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:33537 (local address 127.28.108.190:33641)
0807 08:42:46.633873 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.633876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.634198 (+ 322us) server_negotiation.cc:407] Connection header received
0807 08:42:46.635322 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.635324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.635338 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.635383 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.636303 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.637096 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.638292 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.638590 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.638650 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.651486 (+ 12836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.651489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.651489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.651500 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.655143 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.656063 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.656064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.656065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.656142 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.657152 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.657153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.657153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.657320 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.657371 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.657945 (+ 574us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.658034 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 08:42:46.658236 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.633800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:34167 (local address 127.28.108.188:37301)
0807 08:42:46.634186 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.634189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.634352 (+ 163us) server_negotiation.cc:407] Connection header received
0807 08:42:46.635567 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.635569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.635584 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.635657 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.636317 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.637296 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.638238 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.638580 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.638651 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.652513 (+ 13862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.652514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.652515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.652525 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.655134 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.656154 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.656156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.656156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.656230 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.656533 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.656534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.656534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.656683 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.656766 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.657927 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.658049 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":165,"threads_started":1}
I20260807 08:42:46.658803 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.633795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.129:33537)
0807 08:42:46.634035 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.634081 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.635188 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.635459 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.635463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.635997 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.636181 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.636188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.637178 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.637179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.638208 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.638211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.638716 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.638717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.638808 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.639587 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.639600 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.651291 (+ 11691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.655329 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.655330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.655331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.655947 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.656247 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.656248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.656248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.657058 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.657423 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.657423 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.657464 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.658520 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.658670 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":18,"thread_start_us":164,"threads_started":1}
I20260807 08:42:46.659379 4302 heartbeater.cc:347] Connected to a master server at 127.28.108.189:39863
I20260807 08:42:46.659505 4302 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:46.659509 4301 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33641
I20260807 08:42:46.659580 4300 heartbeater.cc:347] Connected to a master server at 127.28.108.188:37301
I20260807 08:42:46.659603 4301 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:46.659619 4300 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:46.659789 4301 heartbeater.cc:511] Master 127.28.108.190:33641 requested a full tablet report, sending...
I20260807 08:42:46.659787 4302 heartbeater.cc:511] Master 127.28.108.189:39863 requested a full tablet report, sending...
I20260807 08:42:46.659787 4300 heartbeater.cc:511] Master 127.28.108.188:37301 requested a full tablet report, sending...
I20260807 08:42:46.660368 4106 ts_manager.cc:194] Registered new tserver with Master: 9ffc76d5ad2f42879ee70fbe8baf1cb3 (127.28.108.129:37387)
I20260807 08:42:46.660365 3910 ts_manager.cc:194] Registered new tserver with Master: 9ffc76d5ad2f42879ee70fbe8baf1cb3 (127.28.108.129:37387)
I20260807 08:42:46.660600 3260 ts_manager.cc:194] Re-registered known tserver with Master: 9ffc76d5ad2f42879ee70fbe8baf1cb3 (127.28.108.129:37387)
I20260807 08:42:46.661067 3260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:33537
W20260807 08:42:46.751376 4308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:46.751626 4308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:46.751654 4308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:46.753024 4308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:46.753098 4308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:46.753129 4308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:46.756448 4308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab
--principal=oryx/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:41927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=37111
--tserver_master_addrs=127.28.108.190:33641,127.28.108.189:39863,127.28.108.188:37301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:46.756855 4308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:46.757185 4308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:46.763249 4318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:46.763264 4322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:46.763526 4308 server_base.cc:1034] running on GCE node
W20260807 08:42:46.763264 4319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:46.764010 4308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:46.764513 4308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:46.765739 4308 hybrid_clock.cc:648] HybridClock initialized: now 1786092166765669 us; error 79 us; skew 500 ppm
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:46.769243 4308 init.cc:377] Logged in from keytab as oryx/127.28.108.130@KRBTEST.COM (short username oryx)
I20260807 08:42:46.770030 4308 webserver.cc:460] Webserver started at http://127.28.108.130:37111/ using document root <none> and password file <none>
I20260807 08:42:46.770344 4308 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:46.770442 4308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:46.772106 4308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:42:46.772984 4329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.773377 4308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:46.773492 4308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "c58bb082ae12456686f0cd99fced750c"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "b99d218089f41897304af7cd574600fe"
server_key_iv: "8ca3237201af2a0279288ecc13a0c8d0"
server_key_version: "encryptionkey@0"
I20260807 08:42:46.773957 4308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:46.784950 3761 heartbeater.cc:503] Master 127.28.108.190:33641 was elected leader, sending a full tablet report...
I20260807 08:42:46.786275 4308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:46.788564 4308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:46.788714 4308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:46.789018 4308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:46.789393 4308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:46.789455 4308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.789502 4308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:46.789536 4308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.796579 4308 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:41927
I20260807 08:42:46.796665 4450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:41927 every 8 connection(s)
I20260807 08:42:46.797086 4308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:46.805095 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 4308
I20260807 08:42:46.805330 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 3619
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.189@KRBTEST.COM
I20260807 08:42:46.811388 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.798533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:36285 (local address 127.28.108.189:39863)
0807 08:42:46.798616 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.798619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.798629 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:46.799756 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.799758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.799771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.799805 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.800663 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.801477 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.803618 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.803881 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.803941 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.807106 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.807108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.807109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.807120 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.809325 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.810133 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.810135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.810135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.810204 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.810526 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.810527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.810528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.810697 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.810766 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.811171 (+ 405us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.811302 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 08:42:46.812040 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.798347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:39863 (local address 127.28.108.130:36285)
0807 08:42:46.798459 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.798469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.799620 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.799908 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.799909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.800294 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.800549 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.800556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.802835 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.802836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.803496 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.803498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.803964 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.803965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.804035 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.804436 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.804441 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.806912 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.809448 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.809464 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.809468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.809986 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.810287 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.810288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.810289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.810451 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.810802 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.810809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.811011 (+ 202us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.811666 (+ 655us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.811848 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":33,"thread_start_us":71,"threads_started":1}
I20260807 08:42:46.812067 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.798300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:60275 (local address 127.28.108.190:33641)
0807 08:42:46.798373 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.798375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.798476 (+ 101us) server_negotiation.cc:407] Connection header received
0807 08:42:46.799745 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.799748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.799763 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.799800 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.800666 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.801484 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.803281 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.803505 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.803636 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.808372 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.808374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.808375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.808384 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.810324 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.810832 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.810834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.810834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.810907 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.811138 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.811139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.811140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.811306 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.811357 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.811792 (+ 435us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.811904 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 08:42:46.812589 4453 heartbeater.cc:347] Connected to a master server at 127.28.108.189:39863
I20260807 08:42:46.812737 4453 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:46.812881 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.798016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.130:60275)
0807 08:42:46.798401 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.798411 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.799620 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.799919 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.799920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.800332 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.800549 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.800556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.801570 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.801571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.802722 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.802725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.803625 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.803626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.803693 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.804149 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.804159 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.808019 (+ 3860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.810440 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.810440 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.810441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.810725 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.810992 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.810993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.810993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.811062 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.811412 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.811413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.811642 (+ 229us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.812493 (+ 851us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.812660 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":55,"thread_start_us":101,"threads_started":1}
I20260807 08:42:46.812999 4453 heartbeater.cc:511] Master 127.28.108.189:39863 requested a full tablet report, sending...
I20260807 08:42:46.813282 4452 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33641
I20260807 08:42:46.813355 4452 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:46.813447 4452 heartbeater.cc:511] Master 127.28.108.190:33641 requested a full tablet report, sending...
I20260807 08:42:46.813536 3910 ts_manager.cc:194] Registered new tserver with Master: c58bb082ae12456686f0cd99fced750c (127.28.108.130:41927)
I20260807 08:42:46.813736 3260 ts_manager.cc:194] Re-registered known tserver with Master: c58bb082ae12456686f0cd99fced750c (127.28.108.130:41927)
I20260807 08:42:46.814160 3260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:60275
I20260807 08:42:46.816440 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:33891
--local_ip_for_outbound_sockets=127.28.108.131
--tserver_master_addrs=127.28.108.190:33641
--webserver_port=42925
--webserver_interface=127.28.108.131
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab
--principal=oryx/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.108.190:33641,127.28.108.189:39863
--tserver_master_addrs=127.28.108.190:33641,127.28.108.189:39863,127.28.108.188:37301 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.188@KRBTEST.COM
I20260807 08:42:46.822067 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.798077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:46291 (local address 127.28.108.188:37301)
0807 08:42:46.798138 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.798141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.798346 (+ 205us) server_negotiation.cc:407] Connection header received
0807 08:42:46.799724 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.799725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.799748 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.799805 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.800664 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.801198 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.802852 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.803113 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.803211 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.818284 (+ 15073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.818287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.818287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.818298 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.820240 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.820873 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.820876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.820876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.820942 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.821283 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.821284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.821284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.821450 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.821527 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.821824 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.821976 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 08:42:46.822150 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.798024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:37301 (local address 127.28.108.130:46291)
0807 08:42:46.798205 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.798260 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.799586 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.799896 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.799900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.800275 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.800548 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.800553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.801331 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.801333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.802722 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.802725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.803228 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.803229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.803389 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.804452 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.804456 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.818131 (+ 13675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.820453 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.820455 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.820456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.820763 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.821021 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.821022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.821022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.821135 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.821558 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.821559 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.821652 (+ 93us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.821744 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.821912 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":30,"thread_start_us":100,"threads_started":1}
I20260807 08:42:46.823185 4451 heartbeater.cc:347] Connected to a master server at 127.28.108.188:37301
I20260807 08:42:46.823275 4451 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:46.823426 4451 heartbeater.cc:511] Master 127.28.108.188:37301 requested a full tablet report, sending...
I20260807 08:42:46.823834 4106 ts_manager.cc:194] Registered new tserver with Master: c58bb082ae12456686f0cd99fced750c (127.28.108.130:41927)
W20260807 08:42:46.910779 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:46.911007 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:46.911036 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:46.912292 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:46.912341 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:46.912370 4459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:46.914109 4459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab
--principal=oryx/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:33891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=42925
--tserver_master_addrs=127.28.108.190:33641,127.28.108.189:39863,127.28.108.188:37301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:46.914415 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:46.914759 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:46.920517 4466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:46.920598 4465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:46.920619 4468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:46.920972 4459 server_base.cc:1034] running on GCE node
I20260807 08:42:46.921242 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:46.921554 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:46.922794 4459 hybrid_clock.cc:648] HybridClock initialized: now 1786092166922763 us; error 44 us; skew 500 ppm
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:46.926592 4459 init.cc:377] Logged in from keytab as oryx/127.28.108.131@KRBTEST.COM (short username oryx)
I20260807 08:42:46.927312 4459 webserver.cc:460] Webserver started at http://127.28.108.131:42925/ using document root <none> and password file <none>
I20260807 08:42:46.927614 4459 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:46.927711 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:46.929591 4459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260807 08:42:46.930699 4476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.931169 4459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:46.931331 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "7f78c58afe6a423a990b1d498b6a7939"
format_stamp: "Formatted at 2026-08-07 08:42:45 on dist-test-slave-9zdj"
server_key: "15e019ce7e2ec749e949d1986d0098b9"
server_key_iv: "98f5e5d4d59a7dd877278f30dd0c24d2"
server_key_version: "encryptionkey@0"
I20260807 08:42:46.931775 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:46.949147 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:46.951725 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:46.951898 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:46.952183 4459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:46.952569 4459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:46.952629 4459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.952667 4459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:46.952689 4459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:46.957280 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.952856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.28.108.188:37301)
0807 08:42:46.952979 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.952981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.952990 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:42:46.953027 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.953028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.953058 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.953130 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:46.953445 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.954300 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.955989 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.956862 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.956977 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.957082 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.957137 (+ 55us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.957191 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 08:42:46.957280 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.952738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:37301 (local address 127.0.0.1:46158)
0807 08:42:46.952807 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.952820 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.952884 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.953280 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.953282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.953285 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:46.953340 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.953342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.954419 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.954421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.955850 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.955856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.956976 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.956977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.957054 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.957059 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.957128 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.957191 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 08:42:46.957752 4118 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260807 08:42:46.958004 4151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed8bac22a00c4500bdc8eeebc76d77a0. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.958163 4151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: This master's current role is: LEARNER
I20260807 08:42:46.960815 4459 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:33891
I20260807 08:42:46.960875 4598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:33891 every 8 connection(s)
I20260807 08:42:46.961268 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:46.965260 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 4459
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.189@KRBTEST.COM
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.188@KRBTEST.COM
Aug 07 08:42:46 dist-test-slave-9zdj krb5kdc[3213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092166, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:46.977900 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.967339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.28.108.189:39863)
0807 08:42:46.968251 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.968253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.968487 (+ 234us) server_negotiation.cc:407] Connection header received
0807 08:42:46.968511 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.968512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.968525 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.968583 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.969814 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.970598 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.971803 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.972179 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.972244 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.973990 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.973992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.973992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.974002 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.976008 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.976519 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.976521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.976521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.976587 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.976799 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.976800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.976800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.977013 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.977115 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.977594 (+ 479us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.977720 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":59,"threads_started":1}
I20260807 08:42:46.979144 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.962722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:37537 (local address 127.28.108.189:39863)
0807 08:42:46.962772 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.962774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.964192 (+ 1418us) server_negotiation.cc:407] Connection header received
0807 08:42:46.964255 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.964256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.964269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.964319 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.965419 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.966252 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.967961 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.968287 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.968379 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.973267 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.973269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.973269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.973277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.977385 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.977877 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.977878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.977878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.977937 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.978276 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.978277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.978277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.978437 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.978502 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.978949 (+ 447us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.979046 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 08:42:46.979573 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.962656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:39863 (local address 127.28.108.131:37537)
0807 08:42:46.964119 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.964125 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.964192 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.964338 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.964346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.964942 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.965283 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.965290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.966383 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.966383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.967829 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.967832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.968445 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.968446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.968625 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.969499 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.969506 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.973141 (+ 3635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.977480 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.977481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.977482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.977783 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.978015 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.978016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.978017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.978198 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.978532 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.978540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.978786 (+ 246us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.978914 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.979123 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1445,"mutex_wait_us":104,"thread_start_us":73,"threads_started":1}
I20260807 08:42:46.980152 4601 heartbeater.cc:347] Connected to a master server at 127.28.108.189:39863
I20260807 08:42:46.980280 4601 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:46.980517 4601 heartbeater.cc:511] Master 127.28.108.189:39863 requested a full tablet report, sending...
I20260807 08:42:46.980528 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.967852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.28.108.188:37301)
0807 08:42:46.968310 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.968312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.971073 (+ 2761us) server_negotiation.cc:407] Connection header received
0807 08:42:46.971105 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.971106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.971122 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.971180 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.971924 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.972744 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.974353 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.974703 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.974781 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.976498 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.976500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.976500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.976510 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.978626 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.979355 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.979357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.979357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.979426 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.979662 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.979662 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.979663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.979814 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.979927 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.980290 (+ 363us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.980405 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":92,"threads_started":1}
I20260807 08:42:46.981081 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.962315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:37719 (local address 127.28.108.190:33641)
0807 08:42:46.962419 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.962421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.962728 (+ 307us) server_negotiation.cc:407] Connection header received
0807 08:42:46.963974 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.963976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.963990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.964090 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.965445 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.966282 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.967952 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.968275 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.968332 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.973984 (+ 5652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.973986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.973986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.973994 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.976559 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.980145 (+ 3586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.980147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.980147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.980221 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.980467 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.980468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.980469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.980615 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.980663 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.980900 (+ 237us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.980988 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 08:42:46.981099 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.962230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.131:37719)
0807 08:42:46.962573 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.962635 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.963817 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.964381 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.964382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.964945 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.965282 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.965289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.966402 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.966403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.967826 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.967830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.968434 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.968436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.968625 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.969454 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.969469 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.973871 (+ 4402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.977520 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.977522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.977522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.980074 (+ 2552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.980310 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.980312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.980312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.980388 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.980717 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.980718 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.980767 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.980869 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.980989 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":101,"thread_start_us":226,"threads_started":1}
I20260807 08:42:46.981205 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.962312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:55001 (local address 127.28.108.188:37301)
0807 08:42:46.962372 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.962375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.964066 (+ 1691us) server_negotiation.cc:407] Connection header received
0807 08:42:46.964100 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.964101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.964133 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.964217 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.965525 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.966336 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.967951 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.968275 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.968326 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.974661 (+ 6335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.974663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.974663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.974672 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.976925 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.980228 (+ 3303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.980229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.980229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.980300 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.980535 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.980536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.980537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.980687 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.980752 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.981037 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.981132 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 08:42:46.981256 3910 ts_manager.cc:194] Registered new tserver with Master: 7f78c58afe6a423a990b1d498b6a7939 (127.28.108.131:33891)
I20260807 08:42:46.981673 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.962243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:37301 (local address 127.28.108.131:55001)
0807 08:42:46.963992 (+ 1749us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.964001 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.964042 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.964339 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.964344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.964954 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:46.965282 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.965289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.966429 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.966430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.967826 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.967830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.968436 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.968437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.968626 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.969640 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:46.969645 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:46.974537 (+ 4892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:46.977019 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.977035 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.977038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.977463 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.980393 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:46.980394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:46.980394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:46.980459 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:46.980853 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:46.980854 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:46.980908 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.981398 (+ 490us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.981526 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1733,"mutex_wait_us":102,"thread_start_us":110,"threads_started":1}
I20260807 08:42:46.981731 4600 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33641
I20260807 08:42:46.981767 4600 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:46.981838 4600 heartbeater.cc:511] Master 127.28.108.190:33641 requested a full tablet report, sending...
I20260807 08:42:46.982154 3260 ts_manager.cc:194] Re-registered known tserver with Master: 7f78c58afe6a423a990b1d498b6a7939 (127.28.108.131:33891)
I20260807 08:42:46.982519 3260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:37719
I20260807 08:42:46.982527 4599 heartbeater.cc:347] Connected to a master server at 127.28.108.188:37301
I20260807 08:42:46.982597 4599 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:46.982679 4599 heartbeater.cc:511] Master 127.28.108.188:37301 requested a full tablet report, sending...
I20260807 08:42:46.983523 4106 ts_manager.cc:194] Registered new tserver with Master: 7f78c58afe6a423a990b1d498b6a7939 (127.28.108.131:33891)
I20260807 08:42:46.984897 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.966900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.28.108.190:33641)
0807 08:42:46.968784 (+ 1884us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.968787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.970697 (+ 1910us) server_negotiation.cc:407] Connection header received
0807 08:42:46.970733 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.970734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.970750 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.970823 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:46.971633 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.972442 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.978282 (+ 5840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.978648 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.978713 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.981577 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:46.981578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:46.981578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:46.981585 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:46.983512 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.984055 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.984057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.984058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.984160 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:46.984451 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:46.984452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:46.984452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:46.984567 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:46.984681 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.984743 (+ 62us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.984820 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1848,"thread_start_us":103,"threads_started":1}
I20260807 08:42:46.986639 3260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:46.988365 3260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:46.989253 4072 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: NON_VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: true } } }
I20260807 08:42:46.989451 4072 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:46.990928 4072 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0: attempting to promote NON_VOTER 11263b30c20a4565833be137b6e3bd5c to VOTER
I20260807 08:42:46.991063 4072 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } }
I20260807 08:42:46.991559 4119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 LEARNER]: Refusing update from remote peer ed8bac22a00c4500bdc8eeebc76d77a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260807 08:42:46.991632 3919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Refusing update from remote peer ed8bac22a00c4500bdc8eeebc76d77a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260807 08:42:46.991739 4617 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 08:42:46.992350 4617 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 08:42:46.992794 4618 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 11263b30c20a4565833be137b6e3bd5c (127.28.108.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:46.992987 4118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 11263b30c20a4565833be137b6e3bd5c (127.28.108.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:46.993324 3919 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 11263b30c20a4565833be137b6e3bd5c (127.28.108.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:46.994169 4618 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:46.994274 4618 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:46.994347 4618 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:46.994405 4618 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:46.994464 4618 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:46.994516 4618 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:46.994650 4151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:46.994740 4151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:42:46.995060 4626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:46.995168 4626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:42:47.000013 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.995633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.28.108.130:41927)
0807 08:42:46.995827 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.995834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.995847 (+ 13us) server_negotiation.cc:407] Connection header received
0807 08:42:46.995893 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.995895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.995914 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.995998 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:46.996185 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.997303 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.998751 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.999603 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.999711 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.999780 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:46.999872 (+ 92us) server_negotiation.cc:299] Negotiation successful
0807 08:42:46.999916 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":106,"threads_started":1}
I20260807 08:42:46.999997 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.995235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41927 (local address 127.0.0.1:43478)
0807 08:42:46.995283 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.995298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.995342 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.996024 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.996025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.996027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:46.996085 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.996088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.997396 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.997398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.998637 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.998640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.999702 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.999704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.999770 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.999775 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:46.999854 (+ 79us) client_negotiation.cc:240] Negotiation successful
0807 08:42:46.999906 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260807 08:42:47.000262 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.995311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:33891 (local address 127.0.0.1:40806)
0807 08:42:46.995459 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.995469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.995514 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.996866 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.996867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.996870 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:46.996919 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.996920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.998156 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.998157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.999098 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.999101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.999988 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.999989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:47.000074 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:47.000078 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:47.000131 (+ 53us) client_negotiation.cc:240] Negotiation successful
0807 08:42:47.000176 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260807 08:42:47.000537 4630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.995676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.28.108.131:33891)
0807 08:42:46.996669 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.996675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.996687 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:42:46.996727 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.996729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.996745 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.996832 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:46.997023 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.998072 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.999189 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.999753 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.999888 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:46.999952 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:47.000384 (+ 432us) server_negotiation.cc:299] Negotiation successful
0807 08:42:47.000439 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":85,"threads_started":1}
I20260807 08:42:47.001132 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.995342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:37387 (local address 127.0.0.1:41990)
0807 08:42:46.995528 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:46.995541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:46.995601 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:46.996493 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:46.996494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:46.996496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:46.996545 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.996548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.997901 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.997902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:46.999127 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:46.999130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.000871 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.000872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:47.000938 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:47.000943 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:47.001011 (+ 68us) client_negotiation.cc:240] Negotiation successful
0807 08:42:47.001060 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":132,"threads_started":1}
I20260807 08:42:47.001154 4631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:46.995886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.28.108.129:37387)
0807 08:42:46.996244 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:46.996249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:46.996259 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:46.996301 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:46.996303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:46.996319 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:46.996414 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:46.996634 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:46.997806 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:46.999237 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.000069 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.000887 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:47.000955 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:47.001026 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 08:42:47.001072 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":107,"threads_started":1}
I20260807 08:42:47.001060 4379 tablet_service.cc:1467] Processing CreateTablet for tablet 6a24a20b516d4877b09c751cc2486a2e (DEFAULT_TABLE table=test-table [id=baac35d0e0bd45aba1640390bc7cec1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:47.001235 4528 tablet_service.cc:1467] Processing CreateTablet for tablet 6a24a20b516d4877b09c751cc2486a2e (DEFAULT_TABLE table=test-table [id=baac35d0e0bd45aba1640390bc7cec1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:47.001536 4379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a24a20b516d4877b09c751cc2486a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:47.001583 4528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a24a20b516d4877b09c751cc2486a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:47.002274 4230 tablet_service.cc:1467] Processing CreateTablet for tablet 6a24a20b516d4877b09c751cc2486a2e (DEFAULT_TABLE table=test-table [id=baac35d0e0bd45aba1640390bc7cec1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:47.002638 4230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a24a20b516d4877b09c751cc2486a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:47.004746 4632 tablet_bootstrap.cc:492] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: Bootstrap starting.
I20260807 08:42:47.005580 4632 tablet_bootstrap.cc:654] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:47.005966 4633 tablet_bootstrap.cc:492] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: Bootstrap starting.
I20260807 08:42:47.006013 4632 log.cc:824] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:47.006685 4633 tablet_bootstrap.cc:654] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:47.006912 4635 tablet_bootstrap.cc:492] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Bootstrap starting.
I20260807 08:42:47.007032 4633 log.cc:824] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:47.007105 4632 tablet_bootstrap.cc:492] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: No bootstrap required, opened a new log
I20260807 08:42:47.007186 4632 ts_tablet_manager.cc:1397] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:42:47.007767 4635 tablet_bootstrap.cc:654] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:47.007795 4633 tablet_bootstrap.cc:492] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: No bootstrap required, opened a new log
I20260807 08:42:47.007898 4633 ts_tablet_manager.cc:1397] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 08:42:47.008093 4635 log.cc:824] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:47.008956 4635 tablet_bootstrap.cc:492] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: No bootstrap required, opened a new log
I20260807 08:42:47.009039 4635 ts_tablet_manager.cc:1397] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 08:42:47.009960 4632 raft_consensus.cc:348] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:47.010147 4632 raft_consensus.cc:374] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:47.010191 4632 raft_consensus.cc:729] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f78c58afe6a423a990b1d498b6a7939, State: Initialized, Role: FOLLOWER
I20260807 08:42:47.010344 4632 consensus_queue.cc:260] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:47.010457 4633 raft_consensus.cc:348] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:47.010632 4633 raft_consensus.cc:374] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:47.010670 4632 ts_tablet_manager.cc:1428] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:42:47.010692 4633 raft_consensus.cc:729] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58bb082ae12456686f0cd99fced750c, State: Initialized, Role: FOLLOWER
I20260807 08:42:47.010886 4600 heartbeater.cc:503] Master 127.28.108.190:33641 was elected leader, sending a full tablet report...
I20260807 08:42:47.011726 4635 raft_consensus.cc:348] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:47.011911 4635 raft_consensus.cc:374] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:47.011948 4635 raft_consensus.cc:729] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffc76d5ad2f42879ee70fbe8baf1cb3, State: Initialized, Role: FOLLOWER
I20260807 08:42:47.012010 4633 consensus_queue.cc:260] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:47.012087 4635 consensus_queue.cc:260] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:47.012408 4635 ts_tablet_manager.cc:1428] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260807 08:42:47.012540 4633 ts_tablet_manager.cc:1428] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260807 08:42:47.012558 4301 heartbeater.cc:503] Master 127.28.108.190:33641 was elected leader, sending a full tablet report...
I20260807 08:42:47.012909 4452 heartbeater.cc:503] Master 127.28.108.190:33641 was elected leader, sending a full tablet report...
W20260807 08:42:47.048930 4454 tablet.cc:2369] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:42:47.134014 4303 tablet.cc:2369] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:42:47.212401 4602 tablet.cc:2369] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:47.406505 4641 raft_consensus.cc:482] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:47.406651 4641 raft_consensus.cc:504] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:47.407029 4641 leader_election.cc:288] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f78c58afe6a423a990b1d498b6a7939 (127.28.108.131:33891), 9ffc76d5ad2f42879ee70fbe8baf1cb3 (127.28.108.129:37387)
I20260807 08:42:47.410526 4630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:47.407460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:41323 (local address 127.28.108.131:33891)
0807 08:42:47.407553 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:47.407556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:47.407567 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:42:47.407635 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:47.407636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:47.407644 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:47.407684 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:47.407891 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.408428 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.409580 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.410109 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.410176 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:47.410183 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:47.410383 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 08:42:47.410442 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 08:42:47.410555 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:47.407287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:33891 (local address 127.28.108.130:41323)
0807 08:42:47.407478 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:47.407492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:47.407563 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:47.407719 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:47.407720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:47.407722 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:47.407804 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:47.407807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.408534 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.408535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:47.409455 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:47.409458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.410206 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.410207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:47.410285 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:47.410291 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:42:47.410347 (+ 56us) client_negotiation.cc:240] Negotiation successful
0807 08:42:47.410448 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260807 08:42:47.410601 4631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:47.407532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:54625 (local address 127.28.108.129:37387)
0807 08:42:47.407579 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:47.407581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:47.407604 (+ 23us) server_negotiation.cc:407] Connection header received
0807 08:42:47.407637 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:47.407638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:47.407648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:47.407822 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:47.408018 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.408547 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.409739 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.410181 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.410259 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:47.410268 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:47.410473 (+ 205us) server_negotiation.cc:299] Negotiation successful
0807 08:42:47.410523 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:42:47.410921 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:47.407270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:37387 (local address 127.28.108.130:54625)
0807 08:42:47.407418 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:47.407434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:47.407504 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:47.407826 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:47.407827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:47.407829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:47.407917 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:47.407919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.408668 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.408669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:47.409599 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:47.409603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.410283 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.410283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:47.410377 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:47.410382 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:47.410446 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 08:42:47.410488 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260807 08:42:47.411118 4551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a24a20b516d4877b09c751cc2486a2e" candidate_uuid: "c58bb082ae12456686f0cd99fced750c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f78c58afe6a423a990b1d498b6a7939" is_pre_election: true
I20260807 08:42:47.411335 4551 raft_consensus.cc:2454] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c58bb082ae12456686f0cd99fced750c in term 0.
I20260807 08:42:47.411356 4252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a24a20b516d4877b09c751cc2486a2e" candidate_uuid: "c58bb082ae12456686f0cd99fced750c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" is_pre_election: true
I20260807 08:42:47.411550 4252 raft_consensus.cc:2454] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c58bb082ae12456686f0cd99fced750c in term 0.
I20260807 08:42:47.411648 4330 leader_election.cc:302] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f78c58afe6a423a990b1d498b6a7939, c58bb082ae12456686f0cd99fced750c; no voters:
I20260807 08:42:47.411804 4641 raft_consensus.cc:2781] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:47.411875 4641 raft_consensus.cc:482] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:47.411907 4641 raft_consensus.cc:3037] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:47.412670 4641 raft_consensus.cc:504] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:47.412839 4641 leader_election.cc:288] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [CANDIDATE]: Term 1 election: Requested vote from peers 7f78c58afe6a423a990b1d498b6a7939 (127.28.108.131:33891), 9ffc76d5ad2f42879ee70fbe8baf1cb3 (127.28.108.129:37387)
I20260807 08:42:47.412981 4551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a24a20b516d4877b09c751cc2486a2e" candidate_uuid: "c58bb082ae12456686f0cd99fced750c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f78c58afe6a423a990b1d498b6a7939"
I20260807 08:42:47.413085 4551 raft_consensus.cc:3037] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:47.413053 4252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a24a20b516d4877b09c751cc2486a2e" candidate_uuid: "c58bb082ae12456686f0cd99fced750c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3"
I20260807 08:42:47.413172 4252 raft_consensus.cc:3037] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:47.413950 4551 raft_consensus.cc:2454] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c58bb082ae12456686f0cd99fced750c in term 1.
I20260807 08:42:47.414150 4252 raft_consensus.cc:2454] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c58bb082ae12456686f0cd99fced750c in term 1.
I20260807 08:42:47.414196 4330 leader_election.cc:302] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f78c58afe6a423a990b1d498b6a7939, c58bb082ae12456686f0cd99fced750c; no voters:
I20260807 08:42:47.414306 4641 raft_consensus.cc:2781] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:47.414426 4641 raft_consensus.cc:686] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 1 LEADER]: Becoming Leader. State: Replica: c58bb082ae12456686f0cd99fced750c, State: Running, Role: LEADER
I20260807 08:42:47.414546 4641 consensus_queue.cc:237] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:47.415427 3261 catalog_manager.cc:5495] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c reported cstate change: term changed from 0 to 1, leader changed from <none> to c58bb082ae12456686f0cd99fced750c (127.28.108.130). New cstate: current_term: 1 leader_uuid: "c58bb082ae12456686f0cd99fced750c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } health_report { overall_health: UNKNOWN } } }
I20260807 08:42:47.446552 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:47.443562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.28.108.130:41927)
0807 08:42:47.443643 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:47.443646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:47.443656 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:47.443740 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:47.443741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:47.443769 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:47.443854 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:47.444048 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.444601 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.445516 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.445792 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.445885 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:47.446108 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:47.446260 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:47.446389 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:47.446411 (+ 22us) server_negotiation.cc:299] Negotiation successful
0807 08:42:47.446438 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 08:42:47.448793 4252 raft_consensus.cc:1261] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 1 FOLLOWER]: Refusing update from remote peer c58bb082ae12456686f0cd99fced750c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:47.449208 4551 raft_consensus.cc:1261] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939 [term 1 FOLLOWER]: Refusing update from remote peer c58bb082ae12456686f0cd99fced750c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:47.449227 4646 consensus_queue.cc:1035] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:47.449553 4646 consensus_queue.cc:1035] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:47.453023 4649 mvcc.cc:205] Tried to move back new op lower bound from 7315833517867028480 to 7315833517730623488. Current Snapshot: MvccSnapshot[applied={T|T < 7315833517867028480}]
I20260807 08:42:47.453648 4650 mvcc.cc:205] Tried to move back new op lower bound from 7315833517867028480 to 7315833517730623488. Current Snapshot: MvccSnapshot[applied={T|T < 7315833517867028480}]
I20260807 08:42:47.454977 4653 mvcc.cc:205] Tried to move back new op lower bound from 7315833517867028480 to 7315833517730623488. Current Snapshot: MvccSnapshot[applied={T|T < 7315833517867028480}]
I20260807 08:42:47.456832 3261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:47.456945 3261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:47.458629 3261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed8bac22a00c4500bdc8eeebc76d77a0: Sending DeleteTablet for 3 replicas of tablet 6a24a20b516d4877b09c751cc2486a2e
I20260807 08:42:47.459105 4528 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a24a20b516d4877b09c751cc2486a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:47 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:40806
I20260807 08:42:47.459118 4230 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a24a20b516d4877b09c751cc2486a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:47 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:41990
I20260807 08:42:47.459493 4659 tablet_replica.cc:330] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: stopping tablet replica
I20260807 08:42:47.459513 4379 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a24a20b516d4877b09c751cc2486a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:47 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:43478
I20260807 08:42:47.459640 4659 raft_consensus.cc:2229] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:47.459785 4659 raft_consensus.cc:2258] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:47.459920 4660 tablet_replica.cc:330] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: stopping tablet replica
I20260807 08:42:47.460021 4660 raft_consensus.cc:2229] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:47.460413 4659 ts_tablet_manager.cc:1905] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:47.460632 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 3227
I20260807 08:42:47.466226 4659 ts_tablet_manager.cc:1918] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:47.466391 4659 log.cc:1196] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-0/wal/wals/6a24a20b516d4877b09c751cc2486a2e
I20260807 08:42:47.466753 4659 ts_tablet_manager.cc:1939] T 6a24a20b516d4877b09c751cc2486a2e P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Deleting consensus metadata
I20260807 08:42:47.469864 4660 raft_consensus.cc:2258] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:47.470299 4661 tablet_replica.cc:330] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: stopping tablet replica
I20260807 08:42:47.470531 4661 raft_consensus.cc:2229] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:47.470750 4661 raft_consensus.cc:2258] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:47.471442 4661 ts_tablet_manager.cc:1905] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:47.473593 4661 ts_tablet_manager.cc:1918] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:47.473800 4661 log.cc:1196] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/6a24a20b516d4877b09c751cc2486a2e
I20260807 08:42:47.474220 4661 ts_tablet_manager.cc:1939] T 6a24a20b516d4877b09c751cc2486a2e P c58bb082ae12456686f0cd99fced750c: Deleting consensus metadata
W20260807 08:42:47.474750 4182 connection.cc:535] server connection from 127.0.0.1:41990 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 08:42:47.475368 4331 connection.cc:535] server connection from 127.0.0.1:43478 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 08:42:47.476590 4660 ts_tablet_manager.cc:1905] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:47.477751 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:47.460519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46166 (local address 127.28.108.188:37301)
0807 08:42:47.460575 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:47.460577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:47.467407 (+ 6830us) server_negotiation.cc:407] Connection header received
0807 08:42:47.467457 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:47.467458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:47.467474 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:47.467543 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:47.468504 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.469401 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.470820 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.471186 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.471242 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:47.473575 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:47.473577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:47.473577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:47.473588 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:47.475870 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:47.476649 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:47.476650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:47.476651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:47.476733 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:47.477008 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:47.477009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:47.477010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:47.477182 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:47.477294 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:47.477505 (+ 211us) server_negotiation.cc:299] Negotiation successful
0807 08:42:47.477638 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 08:42:47.479621 4660 ts_tablet_manager.cc:1918] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:47.479707 4660 log.cc:1196] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/6a24a20b516d4877b09c751cc2486a2e
I20260807 08:42:47.480017 4660 ts_tablet_manager.cc:1939] T 6a24a20b516d4877b09c751cc2486a2e P 7f78c58afe6a423a990b1d498b6a7939: Deleting consensus metadata
W20260807 08:42:47.480468 4479 connection.cc:441] server connection from 127.0.0.1:40806 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40806 (request call id 1) could send its response
I20260807 08:42:47.489144 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:47.480822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.28.108.189:39863)
0807 08:42:47.481025 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:47.481027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:47.481039 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:42:47.481078 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:47.481079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:47.481097 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:47.481175 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:47.482107 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.482720 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.483600 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.483825 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.483913 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:47.485601 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:47.485603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:47.485604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:47.485718 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:47.487521 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:47.487960 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:47.487962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:47.487962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:47.488033 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:47.488251 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:47.488251 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:47.488252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:47.488423 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:47.488531 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:47.488921 (+ 390us) server_negotiation.cc:299] Negotiation successful
0807 08:42:47.489021 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260807 08:42:47.490712 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:47.480953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.28.108.188:37301)
0807 08:42:47.481006 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:47.481008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:47.481254 (+ 246us) server_negotiation.cc:407] Connection header received
0807 08:42:47.481369 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:47.481370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:47.481381 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:47.481414 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:47.482263 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.483130 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.484745 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:47.485094 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:47.485165 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:47.487478 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:47.487479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:47.487480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:47.487489 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:47.489223 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:47.489683 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:47.489685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:47.489685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:47.489759 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:47.490065 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:47.490066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:47.490066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:47.490222 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:47.490337 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:47.490517 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 08:42:47.490615 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 08:42:48.455960 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.455609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.129:33461)
0807 08:42:48.455787 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.455891 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":129,"threads_started":1}
W20260807 08:42:48.456220 4301 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:33641 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
I20260807 08:42:48.456285 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.455946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.131:56367)
0807 08:42:48.456155 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.456235 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":58,"threads_started":1}
W20260807 08:42:48.456607 4600 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:33641 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
I20260807 08:42:48.457055 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.456781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.130:37635)
0807 08:42:48.456934 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.457007 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":54,"threads_started":1}
W20260807 08:42:48.457276 4452 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:33641 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
I20260807 08:42:48.959043 4673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed8bac22a00c4500bdc8eeebc76d77a0)
I20260807 08:42:48.959200 4673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } }
I20260807 08:42:48.959548 4673 leader_election.cc:288] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641), 71b672a0d5af4ae3b409073cf266f386 (127.28.108.189:39863)
I20260807 08:42:48.959554 4674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed8bac22a00c4500bdc8eeebc76d77a0)
I20260807 08:42:48.959757 4674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } }
I20260807 08:42:48.959988 4675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.959720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49312)
0807 08:42:48.959903 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.959945 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":60,"threads_started":1}
W20260807 08:42:48.960186 4094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
I20260807 08:42:48.960397 4674 leader_election.cc:288] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641), 11263b30c20a4565833be137b6e3bd5c (127.28.108.188:37301)
I20260807 08:42:48.960565 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.960146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49322)
0807 08:42:48.960378 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.960433 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":45,"threads_started":1}
I20260807 08:42:48.960793 4675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.960641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49336)
0807 08:42:48.960729 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.960757 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260807 08:42:48.960811 3890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
W20260807 08:42:48.960891 4094 leader_election.cc:334] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641): Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
I20260807 08:42:48.961326 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.961206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49338)
0807 08:42:48.961263 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.961298 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260807 08:42:48.961498 3890 leader_election.cc:334] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641): Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
I20260807 08:42:48.963960 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.960295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:37301 (local address 127.0.0.1:46194)
0807 08:42:48.960510 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.960522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:48.960583 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:48.960879 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:48.960880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:48.960883 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:48.960939 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:48.960941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:48.961958 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:48.961959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:48.962900 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:48.962903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:48.963570 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:48.963571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:48.963641 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:48.963646 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:48.963732 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 08:42:48.963787 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260807 08:42:48.964090 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.960397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.28.108.188:37301)
0807 08:42:48.960716 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:48.960719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:48.960727 (+ 8us) server_negotiation.cc:407] Connection header received
0807 08:42:48.960751 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:48.960752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:48.960761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:48.960825 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:48.961033 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:48.961864 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:48.962979 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:48.963477 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:48.963575 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:48.963588 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:48.963794 (+ 206us) server_negotiation.cc:299] Negotiation successful
0807 08:42:48.963993 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":82,"threads_started":1}
I20260807 08:42:48.964358 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.959747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:39863 (local address 127.0.0.1:45706)
0807 08:42:48.960000 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.960010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:48.960381 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:48.960624 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:48.960625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:48.960629 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:48.960705 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:48.960707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:48.962232 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:48.962233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:48.963228 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:48.963229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:48.964027 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:48.964028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:48.964104 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:48.964109 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:48.964187 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 08:42:48.964259 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":71,"threads_started":1}
I20260807 08:42:48.964527 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.959827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.28.108.189:39863)
0807 08:42:48.959938 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:48.959940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:48.960344 (+ 404us) server_negotiation.cc:407] Connection header received
0807 08:42:48.960492 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:48.960494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:48.960505 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:48.960543 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:48.961465 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:48.962150 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:48.963346 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:48.963951 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:48.964039 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:48.964052 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:48.964388 (+ 336us) server_negotiation.cc:299] Negotiation successful
0807 08:42:48.964446 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260807 08:42:48.964560 4118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71b672a0d5af4ae3b409073cf266f386" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "11263b30c20a4565833be137b6e3bd5c" is_pre_election: true
I20260807 08:42:48.964720 4118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71b672a0d5af4ae3b409073cf266f386 in term 1.
I20260807 08:42:48.964799 3919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11263b30c20a4565833be137b6e3bd5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "71b672a0d5af4ae3b409073cf266f386" is_pre_election: true
I20260807 08:42:48.964912 3919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11263b30c20a4565833be137b6e3bd5c in term 1.
I20260807 08:42:48.964905 3891 leader_election.cc:302] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11263b30c20a4565833be137b6e3bd5c, 71b672a0d5af4ae3b409073cf266f386; no voters: ed8bac22a00c4500bdc8eeebc76d77a0
I20260807 08:42:48.965054 4674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:42:48.965092 4674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ed8bac22a00c4500bdc8eeebc76d77a0)
I20260807 08:42:48.965112 4674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:42:48.965142 4095 leader_election.cc:302] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11263b30c20a4565833be137b6e3bd5c, 71b672a0d5af4ae3b409073cf266f386; no voters: ed8bac22a00c4500bdc8eeebc76d77a0
I20260807 08:42:48.965257 4673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:42:48.965304 4673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 FOLLOWER]: Starting leader election (detected failure of leader ed8bac22a00c4500bdc8eeebc76d77a0)
I20260807 08:42:48.965341 4673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:42:48.966066 4674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } }
I20260807 08:42:48.966212 4673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } }
I20260807 08:42:48.966261 4674 leader_election.cc:288] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [CANDIDATE]: Term 2 election: Requested vote from peers ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641), 11263b30c20a4565833be137b6e3bd5c (127.28.108.188:37301)
I20260807 08:42:48.966493 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.966371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49340)
0807 08:42:48.966416 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.966457 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260807 08:42:48.966567 4673 leader_election.cc:288] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 2 election: Requested vote from peers ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641), 71b672a0d5af4ae3b409073cf266f386 (127.28.108.189:39863)
I20260807 08:42:48.966697 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.966502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49346)
0807 08:42:48.966617 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.966669 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260807 08:42:48.966720 3919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11263b30c20a4565833be137b6e3bd5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "71b672a0d5af4ae3b409073cf266f386"
I20260807 08:42:48.966575 4118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71b672a0d5af4ae3b409073cf266f386" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "11263b30c20a4565833be137b6e3bd5c"
I20260807 08:42:48.966845 4118 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 71b672a0d5af4ae3b409073cf266f386 in current term 2: Already voted for candidate 11263b30c20a4565833be137b6e3bd5c in this term.
I20260807 08:42:48.966836 3919 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 11263b30c20a4565833be137b6e3bd5c in current term 2: Already voted for candidate 71b672a0d5af4ae3b409073cf266f386 in this term.
I20260807 08:42:48.967190 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.967083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49348)
0807 08:42:48.967128 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.967161 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260807 08:42:48.967280 3890 leader_election.cc:334] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641): Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
I20260807 08:42:48.967334 3890 leader_election.cc:302] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71b672a0d5af4ae3b409073cf266f386; no voters: 11263b30c20a4565833be137b6e3bd5c, ed8bac22a00c4500bdc8eeebc76d77a0
I20260807 08:42:48.967334 4675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:48.967188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49360)
0807 08:42:48.967265 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:48.967303 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 08:42:48.967432 4094 leader_election.cc:334] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641): Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
I20260807 08:42:48.967438 4674 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:42:48.967484 4094 leader_election.cc:302] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11263b30c20a4565833be137b6e3bd5c; no voters: 71b672a0d5af4ae3b409073cf266f386, ed8bac22a00c4500bdc8eeebc76d77a0
I20260807 08:42:48.967555 4673 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 08:42:49.456938 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:49.456604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.129:54405)
0807 08:42:49.456767 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:49.456821 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260807 08:42:49.457353 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:49.456929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.131:39257)
0807 08:42:49.457248 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:49.457295 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":69,"threads_started":1}
I20260807 08:42:49.457799 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:49.457544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.130:47195)
0807 08:42:49.457715 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:49.457752 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":61,"threads_started":1}
I20260807 08:42:50.458161 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.457877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.131:40321)
0807 08:42:50.458057 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.458105 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":62,"threads_started":1}
I20260807 08:42:50.458329 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.457540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.129:41753)
0807 08:42:50.457792 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.457985 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":109,"threads_started":1}
I20260807 08:42:50.458570 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.458189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.130:44683)
0807 08:42:50.458481 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.458520 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":53,"threads_started":1}
I20260807 08:42:50.506896 4692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:50.507010 4692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } }
I20260807 08:42:50.507251 4692 leader_election.cc:288] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641), 71b672a0d5af4ae3b409073cf266f386 (127.28.108.189:39863)
I20260807 08:42:50.507652 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.507441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49594)
0807 08:42:50.507569 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.507609 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260807 08:42:50.507567 3919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11263b30c20a4565833be137b6e3bd5c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "71b672a0d5af4ae3b409073cf266f386" is_pre_election: true
I20260807 08:42:50.507735 3919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11263b30c20a4565833be137b6e3bd5c in term 2.
I20260807 08:42:50.507961 4095 leader_election.cc:302] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11263b30c20a4565833be137b6e3bd5c, 71b672a0d5af4ae3b409073cf266f386; no voters:
I20260807 08:42:50.508092 4692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 08:42:50.508153 4692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:50.508179 4692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:42:50.508230 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.508110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49608)
0807 08:42:50.508157 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.508201 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260807 08:42:50.508327 4094 leader_election.cc:334] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641): Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
I20260807 08:42:50.509006 4692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } }
I20260807 08:42:50.509166 4692 leader_election.cc:288] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 3 election: Requested vote from peers ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641), 71b672a0d5af4ae3b409073cf266f386 (127.28.108.189:39863)
I20260807 08:42:50.509375 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.509265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49614)
0807 08:42:50.509309 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.509347 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260807 08:42:50.509387 3919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11263b30c20a4565833be137b6e3bd5c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "71b672a0d5af4ae3b409073cf266f386"
I20260807 08:42:50.509490 3919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:42:50.509732 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.509652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49630)
0807 08:42:50.509684 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.509716 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260807 08:42:50.509833 4094 leader_election.cc:334] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641): Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
I20260807 08:42:50.510387 3919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11263b30c20a4565833be137b6e3bd5c in term 3.
I20260807 08:42:50.510594 4095 leader_election.cc:302] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11263b30c20a4565833be137b6e3bd5c, 71b672a0d5af4ae3b409073cf266f386; no voters: ed8bac22a00c4500bdc8eeebc76d77a0
I20260807 08:42:50.510708 4692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:42:50.510844 4692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [term 3 LEADER]: Becoming Leader. State: Replica: 11263b30c20a4565833be137b6e3bd5c, State: Running, Role: LEADER
I20260807 08:42:50.510948 4692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } }
I20260807 08:42:50.511415 4695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11263b30c20a4565833be137b6e3bd5c. Latest consensus state: current_term: 3 leader_uuid: "11263b30c20a4565833be137b6e3bd5c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:50.511540 4695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:50.511757 4697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:50.512279 4697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:50.512408 4697 catalog_manager.cc:1259] Loaded cluster ID: 172eadb7b96140c092e0d4aede3b8c81
I20260807 08:42:50.512449 4697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:50.512656 4697 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:50.512761 4697 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: Loaded TSK: 0
I20260807 08:42:50.512951 4697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:42:50.582551 4106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:50.584504 4106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:50.586048 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.585787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49638)
0807 08:42:50.585951 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.585998 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260807 08:42:50.586115 3919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [term 3 FOLLOWER]: Refusing update from remote peer 11263b30c20a4565833be137b6e3bd5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260807 08:42:50.586452 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.586352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49652)
0807 08:42:50.586394 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.586421 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260807 08:42:50.586539 4695 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260807 08:42:50.586743 4094 consensus_peers.cc:487] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c -> Peer ed8bac22a00c4500bdc8eeebc76d77a0 (127.28.108.190:33641): Couldn't send request to peer ed8bac22a00c4500bdc8eeebc76d77a0. Status: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 08:42:50.587464 4700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11263b30c20a4565833be137b6e3bd5c. Latest consensus state: current_term: 3 leader_uuid: "11263b30c20a4565833be137b6e3bd5c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:50.587601 4700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:42:50.588405 4698 mvcc.cc:205] Tried to move back new op lower bound from 7315833530717327360 to 7315833530413670400. Current Snapshot: MvccSnapshot[applied={T|T < 7315833530717327360}]
I20260807 08:42:50.588559 4692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "11263b30c20a4565833be137b6e3bd5c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:50.588733 4692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:50.589146 4700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "11263b30c20a4565833be137b6e3bd5c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed8bac22a00c4500bdc8eeebc76d77a0" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33641 } } peers { permanent_uuid: "71b672a0d5af4ae3b409073cf266f386" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "11263b30c20a4565833be137b6e3bd5c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 37301 } attrs { promote: false } } }
I20260807 08:42:50.589311 4700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b672a0d5af4ae3b409073cf266f386 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:42:50.589314 4701 mvcc.cc:205] Tried to move back new op lower bound from 7315833530717327360 to 7315833530413670400. Current Snapshot: MvccSnapshot[applied={T|T < 7315833530717327360}]
I20260807 08:42:50.595837 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.592430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.28.108.129:37387)
0807 08:42:50.592738 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:50.592740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:50.592747 (+ 7us) server_negotiation.cc:407] Connection header received
0807 08:42:50.592769 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:50.592769 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:50.592779 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:50.592859 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:50.593020 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.593787 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.594907 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.595476 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.595559 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:50.595569 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:50.595709 (+ 140us) server_negotiation.cc:299] Negotiation successful
0807 08:42:50.595758 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":70,"threads_started":1}
I20260807 08:42:50.596004 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.592018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:33891 (local address 127.0.0.1:40814)
0807 08:42:50.592075 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.592088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:50.592140 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:50.593079 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:50.593080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:50.593081 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:50.593113 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:50.593114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.593906 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.593906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:50.595002 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:50.595003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.595772 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.595773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:50.595848 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:50.595853 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:50.595885 (+ 32us) client_negotiation.cc:240] Negotiation successful
0807 08:42:50.595930 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":28}
I20260807 08:42:50.596248 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.592338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:37387 (local address 127.0.0.1:42004)
0807 08:42:50.592460 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.592471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:50.592525 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:50.592863 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:50.592864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:50.592866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:50.592920 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:50.592930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.593867 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.593868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:50.594813 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:50.594815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.595565 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.595566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:50.595616 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:50.595621 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:50.595701 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 08:42:50.595750 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":79,"threads_started":1}
I20260807 08:42:50.596391 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.592084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40814 (local address 127.28.108.131:33891)
0807 08:42:50.592947 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:50.592949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:50.592958 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:42:50.592993 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:50.592994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:50.593007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:50.593063 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:50.593162 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.593793 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.595077 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.595678 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.596206 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:50.596218 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:50.596239 (+ 21us) server_negotiation.cc:299] Negotiation successful
0807 08:42:50.596305 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":82,"threads_started":1}
I20260807 08:42:50.596541 4230 tablet_service.cc:1467] Processing CreateTablet for tablet 6b33b32c068141f28060ce37c3a9296c (DEFAULT_TABLE table=test-table [id=d3a33de759f0431282feefabc5f2769c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:50.596585 4528 tablet_service.cc:1467] Processing CreateTablet for tablet 6b33b32c068141f28060ce37c3a9296c (DEFAULT_TABLE table=test-table [id=d3a33de759f0431282feefabc5f2769c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:50.596714 4230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b33b32c068141f28060ce37c3a9296c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:50.596741 4528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b33b32c068141f28060ce37c3a9296c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:50.596921 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.592137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.28.108.130:41927)
0807 08:42:50.592261 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:50.592264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:50.592466 (+ 202us) server_negotiation.cc:407] Connection header received
0807 08:42:50.592500 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:50.592500 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:50.592513 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:50.592567 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:50.592703 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.593477 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.595611 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.596235 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.596303 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:50.596311 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:50.596766 (+ 455us) server_negotiation.cc:299] Negotiation successful
0807 08:42:50.596837 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260807 08:42:50.597095 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.592019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41927 (local address 127.0.0.1:43496)
0807 08:42:50.592249 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.592260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:50.592408 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:50.592583 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:50.592584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:50.592586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:50.592649 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:50.592651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.593571 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.593572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:50.594809 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:50.594812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:50.596429 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:50.596430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:50.596556 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:50.596562 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:42:50.596802 (+ 240us) client_negotiation.cc:240] Negotiation successful
0807 08:42:50.596864 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":75,"thread_start_us":102,"threads_started":1}
I20260807 08:42:50.597954 4379 tablet_service.cc:1467] Processing CreateTablet for tablet 6b33b32c068141f28060ce37c3a9296c (DEFAULT_TABLE table=test-table [id=d3a33de759f0431282feefabc5f2769c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:50.598099 4379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b33b32c068141f28060ce37c3a9296c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:50.598436 4711 tablet_bootstrap.cc:492] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939: Bootstrap starting.
I20260807 08:42:50.598610 4712 tablet_bootstrap.cc:492] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Bootstrap starting.
I20260807 08:42:50.598852 4711 tablet_bootstrap.cc:654] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:50.598984 4712 tablet_bootstrap.cc:654] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:50.599399 4715 tablet_bootstrap.cc:492] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c: Bootstrap starting.
I20260807 08:42:50.599566 4711 tablet_bootstrap.cc:492] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939: No bootstrap required, opened a new log
I20260807 08:42:50.599625 4711 ts_tablet_manager.cc:1397] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:50.599725 4715 tablet_bootstrap.cc:654] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:50.599828 4711 raft_consensus.cc:348] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:50.599925 4711 raft_consensus.cc:374] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:50.599952 4711 raft_consensus.cc:729] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f78c58afe6a423a990b1d498b6a7939, State: Initialized, Role: FOLLOWER
I20260807 08:42:50.600008 4711 consensus_queue.cc:260] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:50.600211 4711 ts_tablet_manager.cc:1428] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:50.600519 4715 tablet_bootstrap.cc:492] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c: No bootstrap required, opened a new log
I20260807 08:42:50.600580 4715 ts_tablet_manager.cc:1397] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:50.600584 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.600468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.131:49831)
0807 08:42:50.600515 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.600544 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260807 08:42:50.600711 4715 raft_consensus.cc:348] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:50.600787 4715 raft_consensus.cc:374] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:50.600824 4715 raft_consensus.cc:729] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58bb082ae12456686f0cd99fced750c, State: Initialized, Role: FOLLOWER
I20260807 08:42:50.601053 4599 heartbeater.cc:503] Master 127.28.108.188:37301 was elected leader, sending a full tablet report...
I20260807 08:42:50.601231 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.601128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.131:57743)
0807 08:42:50.601166 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.601203 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260807 08:42:50.601383 4712 tablet_bootstrap.cc:492] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3: No bootstrap required, opened a new log
I20260807 08:42:50.601449 4712 ts_tablet_manager.cc:1397] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 08:42:50.600980 4715 consensus_queue.cc:260] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:50.601661 4712 raft_consensus.cc:348] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:50.601737 4712 raft_consensus.cc:374] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:50.601763 4712 raft_consensus.cc:729] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffc76d5ad2f42879ee70fbe8baf1cb3, State: Initialized, Role: FOLLOWER
I20260807 08:42:50.601822 4712 consensus_queue.cc:260] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:50.601905 4715 ts_tablet_manager.cc:1428] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:50.602034 4712 ts_tablet_manager.cc:1428] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:50.602185 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.602050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.130:54867)
0807 08:42:50.602094 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.602129 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260807 08:42:50.602593 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.602370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.129:40351)
0807 08:42:50.602525 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.602560 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260807 08:42:50.602753 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.602656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.130:57325)
0807 08:42:50.602698 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.602723 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260807 08:42:50.603101 4300 heartbeater.cc:503] Master 127.28.108.188:37301 was elected leader, sending a full tablet report...
I20260807 08:42:50.603407 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:50.603272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.129:60461)
0807 08:42:50.603333 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:50.603367 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 08:42:51.033118 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.032947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49668)
0807 08:42:51.033018 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.033064 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260807 08:42:51.033723 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.033573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.0.0.1:49676)
0807 08:42:51.033658 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.033693 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260807 08:42:51.042848 4719 raft_consensus.cc:482] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:51.042852 4717 raft_consensus.cc:482] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:51.043008 4717 raft_consensus.cc:504] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:51.042966 4719 raft_consensus.cc:504] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:51.043493 4717 leader_election.cc:288] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c58bb082ae12456686f0cd99fced750c (127.28.108.130:41927), 9ffc76d5ad2f42879ee70fbe8baf1cb3 (127.28.108.129:37387)
I20260807 08:42:51.043558 4719 leader_election.cc:288] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f78c58afe6a423a990b1d498b6a7939 (127.28.108.131:33891), c58bb082ae12456686f0cd99fced750c (127.28.108.130:41927)
I20260807 08:42:51.047873 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.044000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:43409 (local address 127.28.108.131:33891)
0807 08:42:51.044039 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:51.044061 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:51.044183 (+ 122us) server_negotiation.cc:407] Connection header received
0807 08:42:51.044283 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:51.044284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:51.044293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:51.044325 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:51.044533 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.045334 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.046933 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.047486 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.047544 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:51.047557 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:51.047785 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 08:42:51.047816 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 08:42:51.047873 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.043687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41927 (local address 127.28.108.131:38961)
0807 08:42:51.043753 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.043767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:51.043828 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:51.044376 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:51.044377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:51.044379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:51.044443 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:51.044444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.045452 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.045453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:51.046803 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:51.046807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.047621 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.047622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:51.047685 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:51.047689 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:51.047721 (+ 32us) client_negotiation.cc:240] Negotiation successful
0807 08:42:51.047774 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 08:42:51.048084 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.043687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41927 (local address 127.28.108.129:58057)
0807 08:42:51.044056 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.044070 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:51.044142 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:51.044380 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:51.044382 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:51.044385 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:51.044442 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:51.044445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.045527 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.045528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:51.046900 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:51.046903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.047810 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.047811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:51.047882 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:51.047886 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:51.047924 (+ 38us) client_negotiation.cc:240] Negotiation successful
0807 08:42:51.047969 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344}
I20260807 08:42:51.048177 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.043856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:37387 (local address 127.28.108.131:57897)
0807 08:42:51.044033 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.044044 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:51.044097 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:51.044397 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:51.044398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:51.044400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:51.044478 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:51.044480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.045508 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.045509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:51.047046 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:51.047050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.047904 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.047906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:51.047974 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:51.047977 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:51.048029 (+ 52us) client_negotiation.cc:240] Negotiation successful
0807 08:42:51.048073 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":113,"threads_started":1}
I20260807 08:42:51.048189 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.043722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:33891 (local address 127.28.108.129:43409)
0807 08:42:51.044123 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.044134 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:51.044210 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:51.044403 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:51.044404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:51.044405 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:51.044452 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:51.044455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.045453 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.045453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:51.046803 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:51.046807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.047623 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.047623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:51.047687 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:51.047691 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:51.047999 (+ 308us) client_negotiation.cc:240] Negotiation successful
0807 08:42:51.048058 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":49,"thread_start_us":141,"threads_started":1}
I20260807 08:42:51.048341 4722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.043935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:38961 (local address 127.28.108.130:41927)
0807 08:42:51.044214 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:51.044216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:51.044224 (+ 8us) server_negotiation.cc:407] Connection header received
0807 08:42:51.044265 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:51.044266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:51.044277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:51.044512 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:51.044552 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.045338 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.046934 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.047521 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.048075 (+ 554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:51.048089 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:51.048111 (+ 22us) server_negotiation.cc:299] Negotiation successful
0807 08:42:51.048162 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260807 08:42:51.048449 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.044186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:57897 (local address 127.28.108.129:37387)
0807 08:42:51.044226 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:51.044228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:51.044238 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:51.044266 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:51.044267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:51.044276 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:51.044336 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:51.044562 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.045392 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.047157 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.047760 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.047809 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:51.047820 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:51.048288 (+ 468us) server_negotiation.cc:299] Negotiation successful
0807 08:42:51.048380 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 08:42:51.048481 4551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b33b32c068141f28060ce37c3a9296c" candidate_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f78c58afe6a423a990b1d498b6a7939" is_pre_election: true
I20260807 08:42:51.048578 4551 raft_consensus.cc:2454] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ffc76d5ad2f42879ee70fbe8baf1cb3 in term 0.
I20260807 08:42:51.048586 4401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b33b32c068141f28060ce37c3a9296c" candidate_uuid: "7f78c58afe6a423a990b1d498b6a7939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c58bb082ae12456686f0cd99fced750c" is_pre_election: true
I20260807 08:42:51.048717 4401 raft_consensus.cc:2454] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f78c58afe6a423a990b1d498b6a7939 in term 0.
I20260807 08:42:51.048702 4252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b33b32c068141f28060ce37c3a9296c" candidate_uuid: "7f78c58afe6a423a990b1d498b6a7939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" is_pre_election: true
I20260807 08:42:51.048818 4252 raft_consensus.cc:2454] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f78c58afe6a423a990b1d498b6a7939 in term 0.
I20260807 08:42:51.048920 4479 leader_election.cc:302] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f78c58afe6a423a990b1d498b6a7939, c58bb082ae12456686f0cd99fced750c; no voters:
I20260807 08:42:51.049003 4180 leader_election.cc:302] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f78c58afe6a423a990b1d498b6a7939, 9ffc76d5ad2f42879ee70fbe8baf1cb3; no voters:
I20260807 08:42:51.049129 4717 raft_consensus.cc:2781] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:51.049167 4717 raft_consensus.cc:482] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:51.049165 4719 raft_consensus.cc:2781] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:51.049194 4717 raft_consensus.cc:3037] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:51.049202 4719 raft_consensus.cc:482] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:51.049235 4719 raft_consensus.cc:3037] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:51.049337 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.044026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:58057 (local address 127.28.108.130:41927)
0807 08:42:51.044245 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:51.044247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:51.044256 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:42:51.044280 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:51.044281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:51.044291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:51.044518 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:51.044558 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.045406 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.047040 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.047692 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.049043 (+ 1351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:51.049058 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:51.049081 (+ 23us) server_negotiation.cc:299] Negotiation successful
0807 08:42:51.049153 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":116,"threads_started":1}
I20260807 08:42:51.049520 4401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b33b32c068141f28060ce37c3a9296c" candidate_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c58bb082ae12456686f0cd99fced750c" is_pre_election: true
I20260807 08:42:51.049638 4401 raft_consensus.cc:2454] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ffc76d5ad2f42879ee70fbe8baf1cb3 in term 0.
I20260807 08:42:51.050320 4717 raft_consensus.cc:504] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:51.050465 4717 leader_election.cc:288] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [CANDIDATE]: Term 1 election: Requested vote from peers c58bb082ae12456686f0cd99fced750c (127.28.108.130:41927), 9ffc76d5ad2f42879ee70fbe8baf1cb3 (127.28.108.129:37387)
I20260807 08:42:51.050422 4719 raft_consensus.cc:504] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:51.050676 4401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b33b32c068141f28060ce37c3a9296c" candidate_uuid: "7f78c58afe6a423a990b1d498b6a7939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c58bb082ae12456686f0cd99fced750c"
I20260807 08:42:51.050742 4252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b33b32c068141f28060ce37c3a9296c" candidate_uuid: "7f78c58afe6a423a990b1d498b6a7939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3"
I20260807 08:42:51.050778 4401 raft_consensus.cc:3037] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:51.050851 4252 raft_consensus.cc:2379] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f78c58afe6a423a990b1d498b6a7939 in current term 1: Already voted for candidate 9ffc76d5ad2f42879ee70fbe8baf1cb3 in this term.
I20260807 08:42:51.050822 4551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b33b32c068141f28060ce37c3a9296c" candidate_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f78c58afe6a423a990b1d498b6a7939"
I20260807 08:42:51.050899 4551 raft_consensus.cc:2379] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ffc76d5ad2f42879ee70fbe8baf1cb3 in current term 1: Already voted for candidate 7f78c58afe6a423a990b1d498b6a7939 in this term.
I20260807 08:42:51.050729 4719 leader_election.cc:288] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7f78c58afe6a423a990b1d498b6a7939 (127.28.108.131:33891), c58bb082ae12456686f0cd99fced750c (127.28.108.130:41927)
I20260807 08:42:51.051261 4400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b33b32c068141f28060ce37c3a9296c" candidate_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c58bb082ae12456686f0cd99fced750c"
I20260807 08:42:51.051673 4182 leader_election.cc:302] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ffc76d5ad2f42879ee70fbe8baf1cb3; no voters: 7f78c58afe6a423a990b1d498b6a7939, c58bb082ae12456686f0cd99fced750c
I20260807 08:42:51.051821 4719 raft_consensus.cc:2726] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 08:42:51.051906 4401 raft_consensus.cc:2454] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f78c58afe6a423a990b1d498b6a7939 in term 1.
I20260807 08:42:51.052407 4479 leader_election.cc:302] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f78c58afe6a423a990b1d498b6a7939, c58bb082ae12456686f0cd99fced750c; no voters: 9ffc76d5ad2f42879ee70fbe8baf1cb3
I20260807 08:42:51.052534 4717 raft_consensus.cc:2781] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:51.052842 4717 raft_consensus.cc:686] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 1 LEADER]: Becoming Leader. State: Replica: 7f78c58afe6a423a990b1d498b6a7939, State: Running, Role: LEADER
I20260807 08:42:51.052942 4717 consensus_queue.cc:237] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } }
I20260807 08:42:51.053856 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.053146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.131:60073)
0807 08:42:51.053774 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.053817 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595}
I20260807 08:42:51.053920 4104 catalog_manager.cc:5495] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f78c58afe6a423a990b1d498b6a7939 (127.28.108.131). New cstate: current_term: 1 leader_uuid: "7f78c58afe6a423a990b1d498b6a7939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f78c58afe6a423a990b1d498b6a7939" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 33891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 } health_report { overall_health: UNKNOWN } } }
I20260807 08:42:51.151504 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.148178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.28.108.131:33891)
0807 08:42:51.148253 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:51.148257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:51.148314 (+ 57us) server_negotiation.cc:407] Connection header received
0807 08:42:51.148449 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:51.148450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:51.148471 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:51.148558 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:51.148813 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.149398 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.150504 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:51.150757 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:51.150844 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:51.151051 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:51.151197 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:51.151293 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:51.151356 (+ 63us) server_negotiation.cc:299] Negotiation successful
0807 08:42:51.151384 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 08:42:51.153003 4252 raft_consensus.cc:1261] T 6b33b32c068141f28060ce37c3a9296c P 9ffc76d5ad2f42879ee70fbe8baf1cb3 [term 1 FOLLOWER]: Refusing update from remote peer 7f78c58afe6a423a990b1d498b6a7939: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:51.153056 4401 raft_consensus.cc:1261] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c [term 1 FOLLOWER]: Refusing update from remote peer 7f78c58afe6a423a990b1d498b6a7939: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:51.153370 4717 consensus_queue.cc:1035] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c58bb082ae12456686f0cd99fced750c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:51.153555 4723 consensus_queue.cc:1035] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffc76d5ad2f42879ee70fbe8baf1cb3" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:51.154520 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.154374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.129:44053)
0807 08:42:51.154428 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.154479 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260807 08:42:51.155224 4451 heartbeater.cc:503] Master 127.28.108.188:37301 was elected leader, sending a full tablet report...
I20260807 08:42:51.156013 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.155811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.131:33175)
0807 08:42:51.155935 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.155967 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260807 08:42:51.156565 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.156298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.129:41281)
0807 08:42:51.156496 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.156533 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260807 08:42:51.156816 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.155701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.130:57045)
0807 08:42:51.155876 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.156770 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260807 08:42:51.157956 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.157842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.130:55319)
0807 08:42:51.157889 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.157918 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260807 08:42:51.158396 4104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:51.158488 4104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:51.160188 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:51.160059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33641 (local address 127.28.108.131:38677)
0807 08:42:51.160114 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.160146 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260807 08:42:51.160254 4104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11263b30c20a4565833be137b6e3bd5c: Sending DeleteTablet for 3 replicas of tablet 6b33b32c068141f28060ce37c3a9296c
I20260807 08:42:51.160543 4528 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b33b32c068141f28060ce37c3a9296c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:51 UTC) from {username='oryx', principal='oryx/127.28.108.188@KRBTEST.COM'} at 127.0.0.1:40814
I20260807 08:42:51.160665 4230 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b33b32c068141f28060ce37c3a9296c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:51 UTC) from {username='oryx', principal='oryx/127.28.108.188@KRBTEST.COM'} at 127.0.0.1:42004
I20260807 08:42:51.160799 4527 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b33b32c068141f28060ce37c3a9296c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:51 UTC) from {username='oryx', principal='oryx/127.28.108.188@KRBTEST.COM'} at 127.0.0.1:40814
I20260807 08:42:51.160823 4739 tablet_replica.cc:330] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939: stopping tablet replica
I20260807 08:42:51.160905 4379 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b33b32c068141f28060ce37c3a9296c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:51 UTC) from {username='oryx', principal='oryx/127.28.108.188@KRBTEST.COM'} at 127.0.0.1:43496
I20260807 08:42:51.160936 4739 raft_consensus.cc:2229] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:51.161079 4739 raft_consensus.cc:2258] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:51.161242 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 4067
I20260807 08:42:51.161264 4741 tablet_replica.cc:330] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c: stopping tablet replica
I20260807 08:42:51.161324 4741 raft_consensus.cc:2229] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:51.163457 4741 raft_consensus.cc:2258] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:51.167328 4741 ts_tablet_manager.cc:1905] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:51.169517 4741 ts_tablet_manager.cc:1918] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:51.169653 4741 log.cc:1196] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/6b33b32c068141f28060ce37c3a9296c
I20260807 08:42:51.169960 4741 ts_tablet_manager.cc:1939] T 6b33b32c068141f28060ce37c3a9296c P c58bb082ae12456686f0cd99fced750c: Deleting consensus metadata
I20260807 08:42:51.169982 4739 ts_tablet_manager.cc:1905] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:51.170912 4096 catalog_manager.cc:4841] TS c58bb082ae12456686f0cd99fced750c (127.28.108.130:41927): tablet 6b33b32c068141f28060ce37c3a9296c (table test-table [id=d3a33de759f0431282feefabc5f2769c]) successfully deleted
I20260807 08:42:51.172946 4739 ts_tablet_manager.cc:1918] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:51.173061 4739 log.cc:1196] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/6b33b32c068141f28060ce37c3a9296c
I20260807 08:42:51.173415 4739 ts_tablet_manager.cc:1939] T 6b33b32c068141f28060ce37c3a9296c P 7f78c58afe6a423a990b1d498b6a7939: Deleting consensus metadata
I20260807 08:42:51.174316 4094 catalog_manager.cc:4841] TS 7f78c58afe6a423a990b1d498b6a7939 (127.28.108.131:33891): tablet 6b33b32c068141f28060ce37c3a9296c (table test-table [id=d3a33de759f0431282feefabc5f2769c]) successfully deleted
W20260807 08:42:51.174515 4094 catalog_manager.cc:4808] TS 7f78c58afe6a423a990b1d498b6a7939 (127.28.108.131:33891): delete failed for tablet 6b33b32c068141f28060ce37c3a9296c because the tablet was not found. No further retry: Not found: Tablet not found: 6b33b32c068141f28060ce37c3a9296c
W20260807 08:42:51.183944 4709 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:51.183603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:37387 (local address 127.0.0.1:42018)
0807 08:42:51.183687 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:51.183699 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:51.183809 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:51.183877 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:37387: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 08:42:51.184149 4096 catalog_manager.cc:4584] TS 9ffc76d5ad2f42879ee70fbe8baf1cb3 (127.28.108.129:37387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b33b32c068141f28060ce37c3a9296c: Network error: Client connection negotiation failed: client connection to 127.28.108.129:37387: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260807 08:42:51.184185 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 4308
I20260807 08:42:51.196535 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 4459
I20260807 08:42:51.205077 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 3767
I20260807 08:42:51.213394 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 3967
2026-08-07T08:42:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7509 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:51 dist-test-slave-9zdj krb5kdc[4744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:51 dist-test-slave-9zdj krb5kdc[4744](info): set up 2 sockets
Aug 07 08:42:51 dist-test-slave-9zdj krb5kdc[4744](info): commencing operation
krb5kdc: starting...
W20260807 08:42:52.390303 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:52 dist-test-slave-9zdj krb5kdc[4744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.190.keytab.
Entry for principal oryx/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.190.keytab.
I20260807 08:42:52.548916 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42087
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.190.keytab
--principal=oryx/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:42087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:52.649430 4757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:52.649740 4757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:52.649771 4757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:52.651180 4757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:52.651239 4757 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 08:42:52.651254 4757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:52.651270 4757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:52.651284 4757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:52.651297 4757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:52.653108 4757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:42087
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.190.keytab
--principal=oryx/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:52.653378 4757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:52.653592 4757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:52.659780 4765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:52.659847 4763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:52.659839 4757 server_base.cc:1034] running on GCE node
W20260807 08:42:52.659790 4762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:52.660532 4757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:52.660905 4757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:52.662087 4757 hybrid_clock.cc:648] HybridClock initialized: now 1786092172661963 us; error 136 us; skew 500 ppm
Aug 07 08:42:52 dist-test-slave-9zdj krb5kdc[4744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092172, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:52.665562 4757 init.cc:377] Logged in from keytab as oryx/127.28.108.190@KRBTEST.COM (short username oryx)
I20260807 08:42:52.666225 4757 webserver.cc:460] Webserver started at http://127.28.108.190:44703/ using document root <none> and password file <none>
I20260807 08:42:52.666500 4757 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:52.666554 4757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:52.666707 4757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:52.667979 4757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "28f98d824b3a416389243999efeb569b"
format_stamp: "Formatted at 2026-08-07 08:42:52 on dist-test-slave-9zdj"
server_key: "fc78902674affe8fe939b7584a4f40c3"
server_key_iv: "0a488f67c206c8eddd58e689b543ad4e"
server_key_version: "encryptionkey@0"
I20260807 08:42:52.668457 4757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "28f98d824b3a416389243999efeb569b"
format_stamp: "Formatted at 2026-08-07 08:42:52 on dist-test-slave-9zdj"
server_key: "fc78902674affe8fe939b7584a4f40c3"
server_key_iv: "0a488f67c206c8eddd58e689b543ad4e"
server_key_version: "encryptionkey@0"
I20260807 08:42:52.670135 4757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260807 08:42:52.671166 4772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:52.671437 4757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:52.671547 4757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "28f98d824b3a416389243999efeb569b"
format_stamp: "Formatted at 2026-08-07 08:42:52 on dist-test-slave-9zdj"
server_key: "fc78902674affe8fe939b7584a4f40c3"
server_key_iv: "0a488f67c206c8eddd58e689b543ad4e"
server_key_version: "encryptionkey@0"
I20260807 08:42:52.671639 4757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:52.679503 4757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:52.682468 4757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:52.682649 4757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:52.688853 4757 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:42087
I20260807 08:42:52.688907 4844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:42087 every 8 connection(s)
I20260807 08:42:52.689328 4757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:52.690120 4845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:52.692729 4845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b: Bootstrap starting.
I20260807 08:42:52.693396 4845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:52.693809 4845 log.cc:824] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:52.694690 4845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b: No bootstrap required, opened a new log
I20260807 08:42:52.696641 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 4757
I20260807 08:42:52.696748 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:52.697002 29106 external_mini_cluster.cc:1428] Setting key d652ba0c5e85d4a5c3139d7260656ae9
I20260807 08:42:52.697278 4845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f98d824b3a416389243999efeb569b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42087 } }
I20260807 08:42:52.697516 4845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:52.697571 4845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f98d824b3a416389243999efeb569b, State: Initialized, Role: FOLLOWER
I20260807 08:42:52.697749 4845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f98d824b3a416389243999efeb569b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42087 } }
I20260807 08:42:52.697857 4845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:52.697901 4845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:52.697958 4845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:52.698813 4845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f98d824b3a416389243999efeb569b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42087 } }
I20260807 08:42:52.698917 4845 leader_election.cc:302] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28f98d824b3a416389243999efeb569b; no voters:
I20260807 08:42:52.699128 4845 leader_election.cc:288] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:52.699206 4850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:52.699404 4850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [term 1 LEADER]: Becoming Leader. State: Replica: 28f98d824b3a416389243999efeb569b, State: Running, Role: LEADER
I20260807 08:42:52.699455 4845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:52.699527 4850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f98d824b3a416389243999efeb569b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42087 } }
I20260807 08:42:52.699950 4850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f98d824b3a416389243999efeb569b. Latest consensus state: current_term: 1 leader_uuid: "28f98d824b3a416389243999efeb569b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f98d824b3a416389243999efeb569b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42087 } } }
I20260807 08:42:52.700047 4850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:52.699963 4852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28f98d824b3a416389243999efeb569b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f98d824b3a416389243999efeb569b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42087 } } }
I20260807 08:42:52.700263 4852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:52.700295 4859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:52.700969 4859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:52.702780 4859 catalog_manager.cc:1347] Generated new cluster ID: 214950be159140a4938a994823bfeb9b
I20260807 08:42:52.702842 4859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:42:52 dist-test-slave-9zdj krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:52.712587 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:52.697775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.28.108.190:42087)
0807 08:42:52.698026 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:52.698037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:52.698076 (+ 39us) server_negotiation.cc:407] Connection header received
0807 08:42:52.698221 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:52.698228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:52.698289 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:52.698451 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:52.699496 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:52.701172 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:52.702492 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:52.703147 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:52.703272 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:52.706720 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:52.706727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:52.706728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:52.706746 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:52.709927 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:52.710447 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:52.710452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:52.710452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:52.710615 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:52.710896 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:52.710897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:52.710897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:52.711345 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:52.711543 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:52.712104 (+ 561us) server_negotiation.cc:299] Negotiation successful
0807 08:42:52.712283 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260807 08:42:52.718611 4859 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:52.719533 4859 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:52.725749 4859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b: Generated new TSK 0
I20260807 08:42:52.726042 4859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab.
Entry for principal oryx/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab.
I20260807 08:42:52.766333 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42087
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab
--principal=oryx/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:52.860080 4873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:52.860424 4873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:52.860479 4873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:52.862274 4873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:52.862387 4873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:52.862430 4873 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 08:42:52.862483 4873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:52.864559 4873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab
--principal=oryx/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.108.190:42087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:52.864921 4873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:52.865276 4873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:52.871757 4878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:52.871719 4880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:52.871728 4882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:52.872035 4873 server_base.cc:1034] running on GCE node
I20260807 08:42:52.872470 4873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:52.872843 4873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:52.874024 4873 hybrid_clock.cc:648] HybridClock initialized: now 1786092172874002 us; error 41 us; skew 500 ppm
Aug 07 08:42:52 dist-test-slave-9zdj krb5kdc[4744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092172, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:52.877948 4873 init.cc:377] Logged in from keytab as oryx/127.28.108.129@KRBTEST.COM (short username oryx)
I20260807 08:42:52.878494 4873 webserver.cc:460] Webserver started at http://127.28.108.129:36169/ using document root <none> and password file <none>
I20260807 08:42:52.878794 4873 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:52.878868 4873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:52.879024 4873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:52.880355 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "cbec3ab06bcd4f5b8191093abb0be09e"
format_stamp: "Formatted at 2026-08-07 08:42:52 on dist-test-slave-9zdj"
server_key: "4a3b7e64962b84eaf59f00d4df27354e"
server_key_iv: "f33fefc9b04dbc53e53ba5dce8ec4287"
server_key_version: "encryptionkey@0"
I20260807 08:42:52.880779 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "cbec3ab06bcd4f5b8191093abb0be09e"
format_stamp: "Formatted at 2026-08-07 08:42:52 on dist-test-slave-9zdj"
server_key: "4a3b7e64962b84eaf59f00d4df27354e"
server_key_iv: "f33fefc9b04dbc53e53ba5dce8ec4287"
server_key_version: "encryptionkey@0"
I20260807 08:42:52.882635 4873 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 08:42:52.883730 4890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:52.884009 4873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:52.884140 4873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "cbec3ab06bcd4f5b8191093abb0be09e"
format_stamp: "Formatted at 2026-08-07 08:42:52 on dist-test-slave-9zdj"
server_key: "4a3b7e64962b84eaf59f00d4df27354e"
server_key_iv: "f33fefc9b04dbc53e53ba5dce8ec4287"
server_key_version: "encryptionkey@0"
I20260807 08:42:52.884215 4873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:52.930169 4873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:52.932955 4873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:52.933195 4873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:52.934116 4873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:52.934187 4873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:52.934235 4873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:52.934283 4873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:52.942694 4873 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:39605
I20260807 08:42:52.942884 5013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:39605 every 8 connection(s)
I20260807 08:42:52.943202 4873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
Aug 07 08:42:52 dist-test-slave-9zdj krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092172, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:52.944829 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 4873
I20260807 08:42:52.944972 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:52.945216 29106 external_mini_cluster.cc:1428] Setting key 6011544ebc01aec0dfb52afef50d1f64
I20260807 08:42:52.947860 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:52.934741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:43041 (local address 127.28.108.190:42087)
0807 08:42:52.934786 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:52.934788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:52.935295 (+ 507us) server_negotiation.cc:407] Connection header received
0807 08:42:52.936098 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:52.936101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:52.936119 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:52.936209 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:52.937295 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:52.938387 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:52.939865 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:52.940290 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:52.940377 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:52.943933 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:52.943934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:52.943935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:52.943944 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:52.945887 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:52.946646 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:52.946648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:52.946648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:52.946714 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:52.947043 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:52.947044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:52.947044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:52.947204 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:52.947265 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:52.947650 (+ 385us) server_negotiation.cc:299] Negotiation successful
0807 08:42:52.947744 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 08:42:52.948107 4900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:52.934780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42087 (local address 127.28.108.129:43041)
0807 08:42:52.935045 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:52.935106 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:52.935923 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:52.936305 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:52.936310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:52.936798 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:52.937152 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:52.937160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:52.938523 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:52.938525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:52.939725 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:52.939729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:52.940419 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:52.940420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:52.940570 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:52.941513 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:52.941530 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:52.943774 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:52.945993 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:52.946008 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:52.946010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:52.946538 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:52.946793 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:52.946794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:52.946795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:52.946968 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:52.947271 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:52.947278 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:52.947475 (+ 197us) client_negotiation.cc:769] Sending connection context
0807 08:42:52.947598 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 08:42:52.947804 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":109,"threads_started":1}
I20260807 08:42:52.948774 5014 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42087
I20260807 08:42:52.948899 5014 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:52.949160 5014 heartbeater.cc:511] Master 127.28.108.190:42087 requested a full tablet report, sending...
I20260807 08:42:52.949935 4792 ts_manager.cc:194] Registered new tserver with Master: cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605)
I20260807 08:42:52.950627 4792 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:43041
WARNING: no policy specified for oryx/127.28.108.130@KRBTEST.COM; defaulting to no policy
I20260807 08:42:52.959018 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:52.949898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42087 (local address 127.28.108.129:58699)
0807 08:42:52.950124 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:52.950134 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:52.950182 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:52.950998 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:52.951000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:52.951370 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:52.951440 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:52.951442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:52.952572 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:52.952573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:52.953344 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:52.953347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:52.953976 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:52.953977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:52.954289 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:52.954941 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:52.954947 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:52.955469 (+ 522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:52.957753 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:52.957754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:52.957754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:52.957954 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:52.958261 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:52.958261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:52.958262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:52.958342 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:52.958695 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:52.958696 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:52.958752 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 08:42:52.958813 (+ 61us) client_negotiation.cc:240] Negotiation successful
0807 08:42:52.958926 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260807 08:42:52.959056 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:52.949955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:58699 (local address 127.28.108.190:42087)
0807 08:42:52.950865 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:52.950867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:52.950877 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:52.950900 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:52.950901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:52.950912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:52.950971 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:52.951551 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:52.952443 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:52.953445 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:52.953861 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:52.953967 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:52.955593 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:52.955595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:52.955595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:52.955605 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:52.957566 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:52.958043 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:52.958045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:52.958045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:52.958123 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:52.958425 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:52.958426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:52.958426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:52.958623 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:52.958707 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:52.958868 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 08:42:52.958974 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab.
Entry for principal oryx/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab.
I20260807 08:42:52.982197 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42087
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab
--principal=oryx/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:53.075942 5023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:53.076200 5023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:53.076229 5023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:53.077497 5023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:53.077548 5023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:53.077570 5023 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 08:42:53.077589 5023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:53.079373 5023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab
--principal=oryx/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.108.190:42087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:53.079638 5023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:53.079921 5023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:53.086369 5031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:53.086601 5029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:53.086369 5028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:53.086724 5023 server_base.cc:1034] running on GCE node
I20260807 08:42:53.087030 5023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:53.087282 5023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:53.088515 5023 hybrid_clock.cc:648] HybridClock initialized: now 1786092173088450 us; error 53 us; skew 500 ppm
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092173, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:53.091934 5023 init.cc:377] Logged in from keytab as oryx/127.28.108.130@KRBTEST.COM (short username oryx)
I20260807 08:42:53.092505 5023 webserver.cc:460] Webserver started at http://127.28.108.130:44833/ using document root <none> and password file <none>
I20260807 08:42:53.092821 5023 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:53.092903 5023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:53.093070 5023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:53.094430 5023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "fac85f2650f94e92a772470ffc4e701e"
format_stamp: "Formatted at 2026-08-07 08:42:53 on dist-test-slave-9zdj"
server_key: "7d1257b41b7ed074b4dc715276fe3fec"
server_key_iv: "5782f404db07cde4ae2311bf89a01ce9"
server_key_version: "encryptionkey@0"
I20260807 08:42:53.094853 5023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "fac85f2650f94e92a772470ffc4e701e"
format_stamp: "Formatted at 2026-08-07 08:42:53 on dist-test-slave-9zdj"
server_key: "7d1257b41b7ed074b4dc715276fe3fec"
server_key_iv: "5782f404db07cde4ae2311bf89a01ce9"
server_key_version: "encryptionkey@0"
I20260807 08:42:53.096495 5023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:42:53.097744 5038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:53.097980 5023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:53.098093 5023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "fac85f2650f94e92a772470ffc4e701e"
format_stamp: "Formatted at 2026-08-07 08:42:53 on dist-test-slave-9zdj"
server_key: "7d1257b41b7ed074b4dc715276fe3fec"
server_key_iv: "5782f404db07cde4ae2311bf89a01ce9"
server_key_version: "encryptionkey@0"
I20260807 08:42:53.098194 5023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:53.109972 5023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:53.112943 5023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:53.113157 5023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:53.114219 5023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:53.114451 5023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:53.114497 5023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:53.114521 5023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092173, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:53.122579 5023 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:36419
I20260807 08:42:53.122741 5164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:36419 every 8 connection(s)
I20260807 08:42:53.123063 5023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:53.126617 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.114880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:51553 (local address 127.28.108.190:42087)
0807 08:42:53.115012 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.115014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.115597 (+ 583us) server_negotiation.cc:407] Connection header received
0807 08:42:53.116226 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.116228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.116246 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.116338 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:53.117489 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.118185 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.119401 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.119702 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.119802 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.122783 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:53.122786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:53.122786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:53.122796 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:53.124605 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.125181 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.125182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.125183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.125268 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.125703 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.125705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.125705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.125904 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:53.125970 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.126460 (+ 490us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.126548 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260807 08:42:53.126837 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.115055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42087 (local address 127.28.108.130:51553)
0807 08:42:53.115429 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.115476 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.116045 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.116437 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.116441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.116976 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:53.117343 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.117350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.118310 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.118311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.119285 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.119287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.119830 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.119831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.119988 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.120669 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:53.120679 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:53.122621 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:53.124784 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:53.124793 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:53.124795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:53.125070 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:53.125376 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:53.125377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:53.125377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:53.125553 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:53.125990 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:53.125998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:53.126157 (+ 159us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.126390 (+ 233us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.126541 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":127,"threads_started":1}
I20260807 08:42:53.127543 5165 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42087
I20260807 08:42:53.127665 5165 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:53.127890 5165 heartbeater.cc:511] Master 127.28.108.190:42087 requested a full tablet report, sending...
I20260807 08:42:53.128399 4795 ts_manager.cc:194] Registered new tserver with Master: fac85f2650f94e92a772470ffc4e701e (127.28.108.130:36419)
I20260807 08:42:53.129024 4795 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:51553
I20260807 08:42:53.129869 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 5023
I20260807 08:42:53.129998 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:42:53.130240 29106 external_mini_cluster.cc:1428] Setting key 57387d9e3154fa5e9ef65b785cd415c6
I20260807 08:42:53.135958 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.129115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:46255 (local address 127.28.108.190:42087)
0807 08:42:53.129280 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.129282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.129331 (+ 49us) server_negotiation.cc:407] Connection header received
0807 08:42:53.129401 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.129401 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.129414 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.129466 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:53.129964 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.130596 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.131362 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.131575 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.131662 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.132756 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:53.132758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:53.132758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:53.132765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:53.134266 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.134761 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.134762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.134763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.134831 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.135068 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.135069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.135069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.135366 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:53.135438 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.135702 (+ 264us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.135852 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260807 08:42:53.135949 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.128989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42087 (local address 127.28.108.130:46255)
0807 08:42:53.129253 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.129267 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.129348 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.129480 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.129480 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.129772 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:53.129864 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.129866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.130708 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.130709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.131262 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.131264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.131663 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.131664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.131733 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.132240 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:53.132245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:53.132666 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:53.134350 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:53.134352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:53.134352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:53.134643 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:53.134911 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:53.134912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:53.134913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:53.134997 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:53.135455 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:53.135457 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:53.135543 (+ 86us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.135639 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.135785 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":317,"threads_started":1}
WARNING: no policy specified for oryx/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab.
Entry for principal oryx/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab.
I20260807 08:42:53.166749 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42087
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab
--principal=oryx/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:53.261956 5175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:53.262224 5175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:53.262255 5175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:53.263535 5175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:53.263592 5175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:53.263614 5175 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 08:42:53.263633 5175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:53.265471 5175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab
--principal=oryx/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.108.190:42087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:53.265779 5175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:53.266004 5175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:53.272396 5181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:53.272396 5183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:53.272397 5186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:53.272934 5175 server_base.cc:1034] running on GCE node
I20260807 08:42:53.273137 5175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:53.273532 5175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:53.274706 5175 hybrid_clock.cc:648] HybridClock initialized: now 1786092173274673 us; error 52 us; skew 500 ppm
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092173, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:53.278575 5175 init.cc:377] Logged in from keytab as oryx/127.28.108.131@KRBTEST.COM (short username oryx)
I20260807 08:42:53.279340 5175 webserver.cc:460] Webserver started at http://127.28.108.131:39181/ using document root <none> and password file <none>
I20260807 08:42:53.279697 5175 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:53.279800 5175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:53.279961 5175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:53.281771 5175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "d528f82452f3410a82e4e12367f0240f"
format_stamp: "Formatted at 2026-08-07 08:42:53 on dist-test-slave-9zdj"
server_key: "d1d3ad8e2498369b29e13c173e4bd167"
server_key_iv: "a08c0106a3c7edf0145949deac42c5ee"
server_key_version: "encryptionkey@0"
I20260807 08:42:53.282308 5175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "d528f82452f3410a82e4e12367f0240f"
format_stamp: "Formatted at 2026-08-07 08:42:53 on dist-test-slave-9zdj"
server_key: "d1d3ad8e2498369b29e13c173e4bd167"
server_key_iv: "a08c0106a3c7edf0145949deac42c5ee"
server_key_version: "encryptionkey@0"
I20260807 08:42:53.284528 5175 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260807 08:42:53.285704 5193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:53.285903 5175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:53.286019 5175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "d528f82452f3410a82e4e12367f0240f"
format_stamp: "Formatted at 2026-08-07 08:42:53 on dist-test-slave-9zdj"
server_key: "d1d3ad8e2498369b29e13c173e4bd167"
server_key_iv: "a08c0106a3c7edf0145949deac42c5ee"
server_key_version: "encryptionkey@0"
I20260807 08:42:53.286130 5175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:53.305488 5175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:53.309012 5175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:53.309162 5175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:53.309823 5175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:53.309872 5175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:53.309897 5175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:53.309947 5175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092173, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:53.318361 5175 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:36739
I20260807 08:42:53.318590 5320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:36739 every 8 connection(s)
I20260807 08:42:53.318889 5175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:53.322099 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.310487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:42775 (local address 127.28.108.190:42087)
0807 08:42:53.310567 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.310574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.310824 (+ 250us) server_negotiation.cc:407] Connection header received
0807 08:42:53.311422 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.311424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.311440 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.311534 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:53.312503 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.313384 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.314922 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.315182 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.315282 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.318494 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:53.318497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:53.318497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:53.318508 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:53.320216 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.320789 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.320791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.320791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.320864 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.321143 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.321144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.321144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.321352 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:53.321431 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.321929 (+ 498us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.322019 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 08:42:53.322359 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.310435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42087 (local address 127.28.108.131:42775)
0807 08:42:53.310662 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.310714 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.311303 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.311637 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.311646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.312009 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:53.312303 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.312310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.313509 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.313510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.314798 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.314803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.315325 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.315327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.315485 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.316190 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:53.316208 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:53.318316 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:53.320326 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:53.320335 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:53.320338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:53.320648 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:53.320938 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:53.320938 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:53.320939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:53.321065 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:53.321440 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:53.321447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:53.321691 (+ 244us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.321887 (+ 196us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.322058 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":136,"threads_started":1}
I20260807 08:42:53.323309 5321 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42087
I20260807 08:42:53.323437 5321 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:53.323669 5321 heartbeater.cc:511] Master 127.28.108.190:42087 requested a full tablet report, sending...
I20260807 08:42:53.324143 4795 ts_manager.cc:194] Registered new tserver with Master: d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739)
I20260807 08:42:53.324493 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 5175
I20260807 08:42:53.324615 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:53.324800 4795 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:42775
I20260807 08:42:53.324854 29106 external_mini_cluster.cc:1428] Setting key fbf987a40eb21cb103cb163d1461fb4d
I20260807 08:42:53.325860 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:42:53.332765 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.324573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42087 (local address 127.28.108.131:43637)
0807 08:42:53.324871 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.324884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.325074 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.325329 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.325330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.325579 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:53.325700 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.325703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.326519 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.326520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.327419 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.327424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.327951 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.327952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.328044 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.328942 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:53.328947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:53.329430 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:53.331487 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:53.331489 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:53.331490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:53.331722 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:53.331986 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:53.331987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:53.331987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:53.332058 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:53.332364 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:53.332365 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:53.332428 (+ 63us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.332505 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.332635 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260807 08:42:53.332755 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.324750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:43637 (local address 127.28.108.190:42087)
0807 08:42:53.324801 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.324803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.325042 (+ 239us) server_negotiation.cc:407] Connection header received
0807 08:42:53.325191 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.325193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.325206 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.325270 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:53.325813 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.326389 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.327580 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.327870 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.327938 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.329560 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:53.329562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:53.329562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:53.329575 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:53.331359 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.331826 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.331828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.331828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.331902 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.332138 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.332139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.332139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.332290 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:53.332372 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.332536 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.332633 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 08:42:53.356515 4849 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:53.348665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.28.108.190:42087)
0807 08:42:53.348728 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.348730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.349015 (+ 285us) server_negotiation.cc:407] Connection header received
0807 08:42:53.349250 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.349251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.349267 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.349336 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:53.350132 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.350763 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.351777 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.351986 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.352049 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.356404 (+ 4355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50742: BlockingRecv error: recv got EOF from 127.0.0.1:50742 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:53.366130 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.357730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.28.108.190:42087)
0807 08:42:53.357795 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.357797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.357809 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:42:53.357843 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.357844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.357856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.357923 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:53.358568 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.359112 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.359900 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.360096 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.360163 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.362998 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:53.363000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:53.363000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:53.363009 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:53.364553 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.364958 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.364959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.364960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.365024 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.365262 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.365263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.365263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.365474 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:53.365577 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.365968 (+ 391us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.366057 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260807 08:42:53.368253 4795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:53.370060 4795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:53.377935 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.373230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:36739 (local address 127.0.0.1:55484)
0807 08:42:53.373584 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.373674 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.373823 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.374292 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.374295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.374298 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:53.374401 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.374403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.375502 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.375503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.376666 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.376669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.377509 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.377510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.377610 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.377675 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.377763 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.377823 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260807 08:42:53.378259 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.373605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.28.108.130:36419)
0807 08:42:53.373880 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.373885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.374153 (+ 268us) server_negotiation.cc:407] Connection header received
0807 08:42:53.374201 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.374204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.374220 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.374303 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:53.374508 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.375689 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.377182 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.377881 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.377975 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.378030 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.378119 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.378157 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":144,"threads_started":1}
I20260807 08:42:53.378641 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.373353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.28.108.129:39605)
0807 08:42:53.373564 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.373570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.374023 (+ 453us) server_negotiation.cc:407] Connection header received
0807 08:42:53.374154 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.374157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.374174 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.374237 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:53.374510 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.375696 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.377176 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.377959 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.378053 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.378113 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.378489 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.378535 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":113,"threads_started":1}
I20260807 08:42:53.378664 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.373744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.28.108.131:36739)
0807 08:42:53.373905 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.373911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.373925 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:42:53.373969 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.373971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.373989 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.374077 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:53.374499 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.375409 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.376760 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.377411 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.377506 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.377576 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.378518 (+ 942us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.378575 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260807 08:42:53.378399 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.373230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:39605 (local address 127.0.0.1:38646)
0807 08:42:53.373944 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.373951 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.374056 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.374332 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.374332 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.374334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:53.374419 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.374422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.375810 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.375810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.377080 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.377081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.378158 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.378159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.378216 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.378220 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.378270 (+ 50us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.378311 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":83,"thread_start_us":93,"threads_started":1}
I20260807 08:42:53.378530 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.373472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:36419 (local address 127.0.0.1:33398)
0807 08:42:53.374028 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.374038 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.374090 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.374301 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.374302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.374305 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:53.374419 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.374423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.375797 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.375799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.377078 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.377081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.377971 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.377972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.378043 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.378048 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.378105 (+ 57us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.378155 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":74,"thread_start_us":73,"threads_started":1}
I20260807 08:42:53.379585 5248 tablet_service.cc:1467] Processing CreateTablet for tablet e460c1573f3e43c18d43287af0d60362 (DEFAULT_TABLE table=test-table [id=daf724378438442097488b1d6dad3c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.379585 4947 tablet_service.cc:1467] Processing CreateTablet for tablet e460c1573f3e43c18d43287af0d60362 (DEFAULT_TABLE table=test-table [id=daf724378438442097488b1d6dad3c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.379951 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e460c1573f3e43c18d43287af0d60362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.379961 5248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e460c1573f3e43c18d43287af0d60362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.382145 5089 tablet_service.cc:1467] Processing CreateTablet for tablet e460c1573f3e43c18d43287af0d60362 (DEFAULT_TABLE table=test-table [id=daf724378438442097488b1d6dad3c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.382596 5089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e460c1573f3e43c18d43287af0d60362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.383530 5350 tablet_bootstrap.cc:492] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f: Bootstrap starting.
I20260807 08:42:53.384052 5351 tablet_bootstrap.cc:492] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: Bootstrap starting.
I20260807 08:42:53.384413 5350 tablet_bootstrap.cc:654] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.384765 5351 tablet_bootstrap.cc:654] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.385025 5350 log.cc:824] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:53.385164 5351 log.cc:824] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:53.385917 5350 tablet_bootstrap.cc:492] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f: No bootstrap required, opened a new log
I20260807 08:42:53.385980 5350 ts_tablet_manager.cc:1397] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 08:42:53.386183 5351 tablet_bootstrap.cc:492] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: No bootstrap required, opened a new log
I20260807 08:42:53.386317 5351 ts_tablet_manager.cc:1397] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:42:53.387041 5354 tablet_bootstrap.cc:492] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e: Bootstrap starting.
I20260807 08:42:53.387765 5354 tablet_bootstrap.cc:654] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.387807 5350 raft_consensus.cc:348] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } }
I20260807 08:42:53.388013 5350 raft_consensus.cc:374] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.388060 5350 raft_consensus.cc:729] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d528f82452f3410a82e4e12367f0240f, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.388140 5354 log.cc:824] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:53.388216 5350 consensus_queue.cc:260] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } }
I20260807 08:42:53.388502 5350 ts_tablet_manager.cc:1428] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260807 08:42:53.388513 5321 heartbeater.cc:503] Master 127.28.108.190:42087 was elected leader, sending a full tablet report...
I20260807 08:42:53.388674 5351 raft_consensus.cc:348] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } }
I20260807 08:42:53.388866 5351 raft_consensus.cc:374] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.388908 5351 raft_consensus.cc:729] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbec3ab06bcd4f5b8191093abb0be09e, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.389022 5351 consensus_queue.cc:260] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } }
I20260807 08:42:53.389199 5354 tablet_bootstrap.cc:492] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e: No bootstrap required, opened a new log
I20260807 08:42:53.389269 5354 ts_tablet_manager.cc:1397] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 08:42:53.389321 5351 ts_tablet_manager.cc:1428] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:42:53.389452 5014 heartbeater.cc:503] Master 127.28.108.190:42087 was elected leader, sending a full tablet report...
I20260807 08:42:53.391147 5354 raft_consensus.cc:348] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } }
I20260807 08:42:53.391286 5354 raft_consensus.cc:374] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.391309 5354 raft_consensus.cc:729] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac85f2650f94e92a772470ffc4e701e, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.391417 5354 consensus_queue.cc:260] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } }
I20260807 08:42:53.391703 5354 ts_tablet_manager.cc:1428] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 08:42:53.391722 5165 heartbeater.cc:503] Master 127.28.108.190:42087 was elected leader, sending a full tablet report...
I20260807 08:42:53.436076 5358 raft_consensus.cc:482] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:53.436267 5358 raft_consensus.cc:504] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } }
I20260807 08:42:53.436713 5358 leader_election.cc:288] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739), cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605)
I20260807 08:42:53.439911 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.437068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:57887 (local address 127.28.108.129:39605)
0807 08:42:53.437122 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.437123 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.437134 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:42:53.437190 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.437191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.437202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.437279 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:53.437454 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.438031 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.439125 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.439513 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.439601 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.439612 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.439787 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.439831 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 08:42:53.439911 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.436957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:39605 (local address 127.28.108.130:57887)
0807 08:42:53.437036 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.437062 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.437119 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.437286 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.437287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.437289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:53.437368 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.437375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.438125 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.438126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.439020 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.439022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.439601 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.439602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.439705 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.439709 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.439780 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.439830 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 08:42:53.440017 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.437030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:49503 (local address 127.28.108.131:36739)
0807 08:42:53.437112 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.437115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.437364 (+ 249us) server_negotiation.cc:407] Connection header received
0807 08:42:53.437462 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.437463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.437474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.437541 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:53.437821 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.438271 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.439211 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.439588 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.439721 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.439731 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.439880 (+ 149us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.439945 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 08:42:53.440269 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.436974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:36739 (local address 127.28.108.130:49503)
0807 08:42:53.437198 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.437207 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.437384 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.437561 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.437562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.437564 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:53.437688 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.437690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.438369 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.438369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.439119 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.439121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.439689 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.439690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.439787 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.439791 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.439844 (+ 53us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.439890 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":140,"threads_started":1}
I20260807 08:42:53.440342 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e460c1573f3e43c18d43287af0d60362" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" is_pre_election: true
I20260807 08:42:53.440546 4967 raft_consensus.cc:2454] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 0.
I20260807 08:42:53.440641 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e460c1573f3e43c18d43287af0d60362" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d528f82452f3410a82e4e12367f0240f" is_pre_election: true
I20260807 08:42:53.440816 5272 raft_consensus.cc:2454] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 0.
I20260807 08:42:53.440824 5042 leader_election.cc:302] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbec3ab06bcd4f5b8191093abb0be09e, fac85f2650f94e92a772470ffc4e701e; no voters:
I20260807 08:42:53.440976 5358 raft_consensus.cc:2781] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:53.441062 5358 raft_consensus.cc:482] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:53.441092 5358 raft_consensus.cc:3037] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.442359 5358 raft_consensus.cc:504] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } }
I20260807 08:42:53.442569 5358 leader_election.cc:288] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 election: Requested vote from peers d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739), cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605)
I20260807 08:42:53.442782 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e460c1573f3e43c18d43287af0d60362" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbec3ab06bcd4f5b8191093abb0be09e"
I20260807 08:42:53.442869 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e460c1573f3e43c18d43287af0d60362" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d528f82452f3410a82e4e12367f0240f"
I20260807 08:42:53.442911 4967 raft_consensus.cc:3037] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.443001 5272 raft_consensus.cc:3037] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.443799 4967 raft_consensus.cc:2454] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 1.
I20260807 08:42:53.444036 5272 raft_consensus.cc:2454] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 1.
I20260807 08:42:53.444049 5042 leader_election.cc:302] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbec3ab06bcd4f5b8191093abb0be09e, fac85f2650f94e92a772470ffc4e701e; no voters:
W20260807 08:42:53.444065 5015 tablet.cc:2369] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:53.444267 5358 raft_consensus.cc:2781] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:53.444486 5358 raft_consensus.cc:686] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 1 LEADER]: Becoming Leader. State: Replica: fac85f2650f94e92a772470ffc4e701e, State: Running, Role: LEADER
I20260807 08:42:53.444605 5358 consensus_queue.cc:237] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } }
I20260807 08:42:53.445905 4792 catalog_manager.cc:5495] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e reported cstate change: term changed from 0 to 1, leader changed from <none> to fac85f2650f94e92a772470ffc4e701e (127.28.108.130). New cstate: current_term: 1 leader_uuid: "fac85f2650f94e92a772470ffc4e701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } health_report { overall_health: HEALTHY } } }
I20260807 08:42:53.454190 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.451460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.28.108.130:36419)
0807 08:42:53.451529 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.451531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.451541 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:53.451615 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.451615 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.451635 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.451725 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:53.451874 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.452415 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.453296 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.453501 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.453590 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.453807 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:53.453931 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:53.454030 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.454049 (+ 19us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.454075 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 08:42:53.456702 5272 raft_consensus.cc:1261] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Refusing update from remote peer fac85f2650f94e92a772470ffc4e701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:53.456713 4967 raft_consensus.cc:1261] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Refusing update from remote peer fac85f2650f94e92a772470ffc4e701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:53.457149 5358 consensus_queue.cc:1035] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:53.457393 5360 consensus_queue.cc:1035] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:53.460564 5365 mvcc.cc:205] Tried to move back new op lower bound from 7315833542474956800 to 7315833542429843456. Current Snapshot: MvccSnapshot[applied={T|T < 7315833542474956800}]
I20260807 08:42:53.463505 4792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:53.463646 4792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:53.464664 4792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b: Sending DeleteTablet for 3 replicas of tablet e460c1573f3e43c18d43287af0d60362
I20260807 08:42:53.465067 5248 tablet_service.cc:1514] Processing DeleteTablet for tablet e460c1573f3e43c18d43287af0d60362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:53 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:55484
I20260807 08:42:53.465093 4947 tablet_service.cc:1514] Processing DeleteTablet for tablet e460c1573f3e43c18d43287af0d60362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:53 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:38646
I20260807 08:42:53.465441 5089 tablet_service.cc:1514] Processing DeleteTablet for tablet e460c1573f3e43c18d43287af0d60362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:53 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:33398
I20260807 08:42:53.465739 5375 tablet_replica.cc:330] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e: stopping tablet replica
I20260807 08:42:53.465768 5373 tablet_replica.cc:330] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: stopping tablet replica
I20260807 08:42:53.465749 4792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 08:42:53.465889 5373 raft_consensus.cc:2229] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260807 08:42:53.465922 4792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:53.466005 5373 raft_consensus.cc:2258] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:53.466152 5375 raft_consensus.cc:2229] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:53.466399 5374 tablet_replica.cc:330] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f: stopping tablet replica
I20260807 08:42:53.466756 5373 ts_tablet_manager.cc:1905] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:53.467228 5375 raft_consensus.cc:2258] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:53.467525 4947 tablet_service.cc:1467] Processing CreateTablet for tablet 1831c93381ca40c08e0c98bf7a7546a9 (DEFAULT_TABLE table=test-table [id=1fff9a0b25d74d2abac6427f301690c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.467665 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1831c93381ca40c08e0c98bf7a7546a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.468300 5375 ts_tablet_manager.cc:1905] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:53.469009 5351 tablet_bootstrap.cc:492] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e: Bootstrap starting.
I20260807 08:42:53.468999 5374 raft_consensus.cc:2229] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:53.469185 5374 raft_consensus.cc:2258] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:53.469455 5351 tablet_bootstrap.cc:654] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.469827 5374 ts_tablet_manager.cc:1905] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:53.470096 5089 tablet_service.cc:1467] Processing CreateTablet for tablet 1831c93381ca40c08e0c98bf7a7546a9 (DEFAULT_TABLE table=test-table [id=1fff9a0b25d74d2abac6427f301690c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.470224 5089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1831c93381ca40c08e0c98bf7a7546a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.471079 5375 ts_tablet_manager.cc:1918] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:53.471236 5375 log.cc:1196] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/e460c1573f3e43c18d43287af0d60362
I20260807 08:42:53.471626 5375 ts_tablet_manager.cc:1939] T e460c1573f3e43c18d43287af0d60362 P fac85f2650f94e92a772470ffc4e701e: Deleting consensus metadata
I20260807 08:42:53.471838 5354 tablet_bootstrap.cc:492] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e: Bootstrap starting.
I20260807 08:42:53.472329 5248 tablet_service.cc:1467] Processing CreateTablet for tablet 1831c93381ca40c08e0c98bf7a7546a9 (DEFAULT_TABLE table=test-table [id=1fff9a0b25d74d2abac6427f301690c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.472347 5373 ts_tablet_manager.cc:1918] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:53.472465 5373 log.cc:1196] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/wal/wals/e460c1573f3e43c18d43287af0d60362
I20260807 08:42:53.472594 5354 tablet_bootstrap.cc:654] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.472831 5373 ts_tablet_manager.cc:1939] T e460c1573f3e43c18d43287af0d60362 P cbec3ab06bcd4f5b8191093abb0be09e: Deleting consensus metadata
I20260807 08:42:53.473284 4776 catalog_manager.cc:4841] TS fac85f2650f94e92a772470ffc4e701e (127.28.108.130:36419): tablet e460c1573f3e43c18d43287af0d60362 (table test-table [id=daf724378438442097488b1d6dad3c37]) successfully deleted
I20260807 08:42:53.473591 4776 catalog_manager.cc:4841] TS cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605): tablet e460c1573f3e43c18d43287af0d60362 (table test-table [id=daf724378438442097488b1d6dad3c37]) successfully deleted
I20260807 08:42:53.473623 5248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1831c93381ca40c08e0c98bf7a7546a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.474294 5354 tablet_bootstrap.cc:492] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e: No bootstrap required, opened a new log
I20260807 08:42:53.474361 5354 ts_tablet_manager.cc:1397] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 08:42:53.474539 5354 raft_consensus.cc:348] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.474646 5354 raft_consensus.cc:374] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.474671 5354 raft_consensus.cc:729] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac85f2650f94e92a772470ffc4e701e, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.474727 5354 consensus_queue.cc:260] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.474861 5354 ts_tablet_manager.cc:1428] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:53.476215 5350 tablet_bootstrap.cc:492] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f: Bootstrap starting.
I20260807 08:42:53.476720 5350 tablet_bootstrap.cc:654] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.477190 5374 ts_tablet_manager.cc:1918] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:53.477298 5374 log.cc:1196] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/e460c1573f3e43c18d43287af0d60362
I20260807 08:42:53.477527 5374 ts_tablet_manager.cc:1939] T e460c1573f3e43c18d43287af0d60362 P d528f82452f3410a82e4e12367f0240f: Deleting consensus metadata
I20260807 08:42:53.478049 4774 catalog_manager.cc:4841] TS d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739): tablet e460c1573f3e43c18d43287af0d60362 (table test-table [id=daf724378438442097488b1d6dad3c37]) successfully deleted
I20260807 08:42:53.478386 5351 tablet_bootstrap.cc:492] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e: No bootstrap required, opened a new log
I20260807 08:42:53.478446 5351 ts_tablet_manager.cc:1397] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260807 08:42:53.478853 5351 raft_consensus.cc:348] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.478952 5351 raft_consensus.cc:374] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.478967 5351 raft_consensus.cc:729] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbec3ab06bcd4f5b8191093abb0be09e, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.479033 5351 consensus_queue.cc:260] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.479161 5351 ts_tablet_manager.cc:1428] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:53.482425 5350 tablet_bootstrap.cc:492] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f: No bootstrap required, opened a new log
I20260807 08:42:53.482501 5350 ts_tablet_manager.cc:1397] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260807 08:42:53.482882 5350 raft_consensus.cc:348] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.482990 5350 raft_consensus.cc:374] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.483006 5350 raft_consensus.cc:729] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d528f82452f3410a82e4e12367f0240f, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.483070 5350 consensus_queue.cc:260] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.483173 5350 ts_tablet_manager.cc:1428] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 08:42:53.496892 5357 raft_consensus.cc:482] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:53.497020 5357 raft_consensus.cc:504] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.497476 5357 leader_election.cc:288] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fac85f2650f94e92a772470ffc4e701e (127.28.108.130:36419), d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739)
I20260807 08:42:53.501564 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.497717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:36739 (local address 127.28.108.129:49069)
0807 08:42:53.497937 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.497950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.498062 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.498258 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.498259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.498261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:53.498342 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.498344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.499365 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.499366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.500429 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.500434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.501257 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.501259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.501341 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.501345 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.501402 (+ 57us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.501452 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260807 08:42:53.501567 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.497925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:49069 (local address 127.28.108.131:36739)
0807 08:42:53.497975 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.497977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.498035 (+ 58us) server_negotiation.cc:407] Connection header received
0807 08:42:53.498173 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.498174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.498182 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.498247 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:53.498413 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.499259 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.500556 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.501078 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.501147 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.501158 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.501433 (+ 275us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.501488 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 08:42:53.501994 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1831c93381ca40c08e0c98bf7a7546a9" candidate_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d528f82452f3410a82e4e12367f0240f" is_pre_election: true
I20260807 08:42:53.502125 5272 raft_consensus.cc:2454] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbec3ab06bcd4f5b8191093abb0be09e in term 0.
I20260807 08:42:53.502363 4892 leader_election.cc:302] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbec3ab06bcd4f5b8191093abb0be09e, d528f82452f3410a82e4e12367f0240f; no voters:
I20260807 08:42:53.502542 5357 raft_consensus.cc:2781] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:53.502565 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.497719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:36419 (local address 127.28.108.129:35961)
0807 08:42:53.497969 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.497979 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.498059 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.498305 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.498307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.498309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:53.498401 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.498403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.499319 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.499320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.501039 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.501043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.502222 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.502223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.502307 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.502312 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.502424 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.502473 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":35,"spinlock_wait_cycles":768,"thread_start_us":145,"threads_started":1}
I20260807 08:42:53.502581 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.497862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:35961 (local address 127.28.108.130:36419)
0807 08:42:53.497916 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.497919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.498026 (+ 107us) server_negotiation.cc:407] Connection header received
0807 08:42:53.498151 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.498153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.498176 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.498245 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:53.498477 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.499217 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.501146 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.501823 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.501907 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.501917 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.502452 (+ 535us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.502508 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 08:42:53.502576 5357 raft_consensus.cc:482] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:53.502986 5357 raft_consensus.cc:3037] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.502975 5113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1831c93381ca40c08e0c98bf7a7546a9" candidate_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac85f2650f94e92a772470ffc4e701e" is_pre_election: true
I20260807 08:42:53.503139 5113 raft_consensus.cc:2454] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbec3ab06bcd4f5b8191093abb0be09e in term 0.
I20260807 08:42:53.503952 5357 raft_consensus.cc:504] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.504101 5357 leader_election.cc:288] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [CANDIDATE]: Term 1 election: Requested vote from peers fac85f2650f94e92a772470ffc4e701e (127.28.108.130:36419), d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739)
I20260807 08:42:53.504261 5113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1831c93381ca40c08e0c98bf7a7546a9" candidate_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac85f2650f94e92a772470ffc4e701e"
I20260807 08:42:53.504333 5113 raft_consensus.cc:3037] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.504560 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1831c93381ca40c08e0c98bf7a7546a9" candidate_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d528f82452f3410a82e4e12367f0240f"
I20260807 08:42:53.504662 5272 raft_consensus.cc:3037] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.505224 5113 raft_consensus.cc:2454] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbec3ab06bcd4f5b8191093abb0be09e in term 1.
I20260807 08:42:53.505436 4895 leader_election.cc:302] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbec3ab06bcd4f5b8191093abb0be09e, fac85f2650f94e92a772470ffc4e701e; no voters:
I20260807 08:42:53.505544 5272 raft_consensus.cc:2454] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbec3ab06bcd4f5b8191093abb0be09e in term 1.
I20260807 08:42:53.505559 5357 raft_consensus.cc:2781] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:53.505751 5357 raft_consensus.cc:686] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 1 LEADER]: Becoming Leader. State: Replica: cbec3ab06bcd4f5b8191093abb0be09e, State: Running, Role: LEADER
I20260807 08:42:53.505841 5357 consensus_queue.cc:237] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.506742 4795 catalog_manager.cc:5495] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e reported cstate change: term changed from 0 to 1, leader changed from <none> to cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129). New cstate: current_term: 1 leader_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } health_report { overall_health: UNKNOWN } } }
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092172, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.190@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:53.539891 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.530662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42087 (local address 127.0.0.1:50766)
0807 08:42:53.530918 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.530935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.531000 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.531199 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.531200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.531578 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:53.531634 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.531637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.532688 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.532689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.533372 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.533375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.533890 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.533891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.533956 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.534676 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:53.534681 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:53.536269 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:53.538440 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:53.538451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:53.538453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:53.538695 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:53.539032 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:53.539033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:53.539033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:53.539124 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:53.539515 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:53.539524 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:53.539580 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.539650 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.539801 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":188,"threads_started":1}
I20260807 08:42:53.540030 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.530758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.28.108.190:42087)
0807 08:42:53.530831 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.530834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.530999 (+ 165us) server_negotiation.cc:407] Connection header received
0807 08:42:53.531079 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.531080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.531093 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.531152 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:53.531744 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.532543 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.533475 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.533804 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.533887 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.536385 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:53.536387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:53.536388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:53.536398 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:53.538312 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.538847 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.538849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.538849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.538927 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.539221 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.539222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.539223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.539412 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:53.539506 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.539708 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.539882 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":40}
I20260807 08:42:53.542953 4795 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:50766:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 08:42:53.543393 4795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:53.545490 5248 tablet_service.cc:1467] Processing CreateTablet for tablet d30fa16ca34d404988585d4ec120a66b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09c91452d3ed43bc957bd4570c658c8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 08:42:53.545542 4947 tablet_service.cc:1467] Processing CreateTablet for tablet d30fa16ca34d404988585d4ec120a66b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09c91452d3ed43bc957bd4570c658c8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 08:42:53.545557 5089 tablet_service.cc:1467] Processing CreateTablet for tablet d30fa16ca34d404988585d4ec120a66b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09c91452d3ed43bc957bd4570c658c8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 08:42:53.545730 5248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30fa16ca34d404988585d4ec120a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.545787 5089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30fa16ca34d404988585d4ec120a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.545879 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30fa16ca34d404988585d4ec120a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.547482 5354 tablet_bootstrap.cc:492] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: Bootstrap starting.
I20260807 08:42:53.547489 5351 tablet_bootstrap.cc:492] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e: Bootstrap starting.
I20260807 08:42:53.547489 5350 tablet_bootstrap.cc:492] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f: Bootstrap starting.
I20260807 08:42:53.547950 5354 tablet_bootstrap.cc:654] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.547963 5350 tablet_bootstrap.cc:654] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.547963 5351 tablet_bootstrap.cc:654] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.549024 5350 tablet_bootstrap.cc:492] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f: No bootstrap required, opened a new log
I20260807 08:42:53.549099 5350 ts_tablet_manager.cc:1397] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 08:42:53.549281 5350 raft_consensus.cc:348] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.549369 5350 raft_consensus.cc:374] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.549396 5350 raft_consensus.cc:729] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d528f82452f3410a82e4e12367f0240f, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.549453 5350 consensus_queue.cc:260] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.549558 5350 ts_tablet_manager.cc:1428] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 08:42:53.549777 5356 tablet_replica.cc:439] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } }
I20260807 08:42:53.549877 5356 tablet_replica.cc:442] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:53.550567 5354 tablet_bootstrap.cc:492] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: No bootstrap required, opened a new log
I20260807 08:42:53.550580 5351 tablet_bootstrap.cc:492] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e: No bootstrap required, opened a new log
I20260807 08:42:53.550665 5351 ts_tablet_manager.cc:1397] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 08:42:53.550665 5354 ts_tablet_manager.cc:1397] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 08:42:53.550837 5351 raft_consensus.cc:348] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.550896 5351 raft_consensus.cc:374] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.550909 5351 raft_consensus.cc:729] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbec3ab06bcd4f5b8191093abb0be09e, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.550973 5351 consensus_queue.cc:260] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.551069 5351 ts_tablet_manager.cc:1428] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:53.551011 5354 raft_consensus.cc:348] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.551115 5354 raft_consensus.cc:374] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.551137 5354 raft_consensus.cc:729] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac85f2650f94e92a772470ffc4e701e, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.551183 5354 consensus_queue.cc:260] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.551155 5382 tablet_replica.cc:439] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } }
I20260807 08:42:53.551254 5382 tablet_replica.cc:442] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:53.551285 5354 ts_tablet_manager.cc:1428] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:53.551437 5368 tablet_replica.cc:439] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } }
I20260807 08:42:53.551529 5368 tablet_replica.cc:442] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 08:42:53.569535 5322 tablet.cc:2369] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:42:53.624028 5166 tablet.cc:2369] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:42:53.741345 5368 raft_consensus.cc:482] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:53.741505 5368 raft_consensus.cc:504] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.741793 5368 leader_election.cc:288] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605), d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739)
I20260807 08:42:53.742273 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30fa16ca34d404988585d4ec120a66b" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d528f82452f3410a82e4e12367f0240f" is_pre_election: true
I20260807 08:42:53.742364 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30fa16ca34d404988585d4ec120a66b" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" is_pre_election: true
I20260807 08:42:53.742427 5272 raft_consensus.cc:2454] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 0.
I20260807 08:42:53.742466 4967 raft_consensus.cc:2454] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 0.
I20260807 08:42:53.742667 5040 leader_election.cc:302] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d528f82452f3410a82e4e12367f0240f, fac85f2650f94e92a772470ffc4e701e; no voters:
I20260807 08:42:53.742841 5368 raft_consensus.cc:2781] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:53.742895 5368 raft_consensus.cc:482] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:53.742911 5368 raft_consensus.cc:3037] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.743707 5368 raft_consensus.cc:504] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.743911 5368 leader_election.cc:288] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 election: Requested vote from peers cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605), d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739)
I20260807 08:42:53.744148 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30fa16ca34d404988585d4ec120a66b" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d528f82452f3410a82e4e12367f0240f"
I20260807 08:42:53.744146 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30fa16ca34d404988585d4ec120a66b" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbec3ab06bcd4f5b8191093abb0be09e"
I20260807 08:42:53.744262 5272 raft_consensus.cc:3037] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.744262 4967 raft_consensus.cc:3037] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.745239 4967 raft_consensus.cc:2454] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 1.
I20260807 08:42:53.745235 5272 raft_consensus.cc:2454] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 1.
I20260807 08:42:53.745549 5042 leader_election.cc:302] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbec3ab06bcd4f5b8191093abb0be09e, fac85f2650f94e92a772470ffc4e701e; no voters:
I20260807 08:42:53.745702 5368 raft_consensus.cc:2781] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:53.745754 5368 raft_consensus.cc:686] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [term 1 LEADER]: Becoming Leader. State: Replica: fac85f2650f94e92a772470ffc4e701e, State: Running, Role: LEADER
I20260807 08:42:53.745812 5368 consensus_queue.cc:237] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.746224 5358 tablet_replica.cc:439] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: TxnStatusTablet state changed. Reason: New leader fac85f2650f94e92a772470ffc4e701e. Latest consensus state: current_term: 1 leader_uuid: "fac85f2650f94e92a772470ffc4e701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } }
I20260807 08:42:53.746342 5358 tablet_replica.cc:442] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: This TxnStatusTablet replica's current role is: LEADER
I20260807 08:42:53.746511 5394 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 08:42:53.746675 5394 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 08:42:53.746817 4791 catalog_manager.cc:5495] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e reported cstate change: term changed from 0 to 1, leader changed from <none> to fac85f2650f94e92a772470ffc4e701e (127.28.108.130). New cstate: current_term: 1 leader_uuid: "fac85f2650f94e92a772470ffc4e701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } health_report { overall_health: UNKNOWN } } }
I20260807 08:42:53.784983 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.782233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.28.108.130:36419)
0807 08:42:53.782290 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.782292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.782322 (+ 30us) server_negotiation.cc:407] Connection header received
0807 08:42:53.782352 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.782352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.782365 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.782418 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:53.782572 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.783098 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.784250 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.784465 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.784528 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.784754 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:53.784787 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:53.784822 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.784892 (+ 70us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.784911 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 08:42:53.785035 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.782085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:36419 (local address 127.0.0.1:33420)
0807 08:42:53.782167 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.782182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.782257 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.782440 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.782441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.782443 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:53.782501 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.782504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.783184 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.783185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.784145 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.784149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.784547 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.784548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.784626 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.784661 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:53.784834 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:53.784840 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.784889 (+ 49us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.784944 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260807 08:42:53.786273 5272 raft_consensus.cc:1261] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Refusing update from remote peer fac85f2650f94e92a772470ffc4e701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:53.786391 4967 raft_consensus.cc:1261] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Refusing update from remote peer fac85f2650f94e92a772470ffc4e701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:53.786607 5368 consensus_queue.cc:1035] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:53.786739 5358 consensus_queue.cc:1035] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:53.787312 5356 tablet_replica.cc:439] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f: TxnStatusTablet state changed. Reason: New leader fac85f2650f94e92a772470ffc4e701e. Latest consensus state: current_term: 1 leader_uuid: "fac85f2650f94e92a772470ffc4e701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } }
I20260807 08:42:53.787429 5356 tablet_replica.cc:442] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:53.787415 5382 tablet_replica.cc:439] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e: TxnStatusTablet state changed. Reason: New leader fac85f2650f94e92a772470ffc4e701e. Latest consensus state: current_term: 1 leader_uuid: "fac85f2650f94e92a772470ffc4e701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } }
I20260807 08:42:53.787510 5382 tablet_replica.cc:442] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:53.788257 5368 tablet_replica.cc:439] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fac85f2650f94e92a772470ffc4e701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } }
I20260807 08:42:53.788354 5368 tablet_replica.cc:442] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: This TxnStatusTablet replica's current role is: LEADER
I20260807 08:42:53.788648 5382 tablet_replica.cc:439] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fac85f2650f94e92a772470ffc4e701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } }
I20260807 08:42:53.788745 5382 tablet_replica.cc:442] T d30fa16ca34d404988585d4ec120a66b P cbec3ab06bcd4f5b8191093abb0be09e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:53.788903 5360 tablet_replica.cc:439] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fac85f2650f94e92a772470ffc4e701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } }
I20260807 08:42:53.788988 5360 tablet_replica.cc:442] T d30fa16ca34d404988585d4ec120a66b P fac85f2650f94e92a772470ffc4e701e: This TxnStatusTablet replica's current role is: LEADER
I20260807 08:42:53.789280 5364 mvcc.cc:205] Tried to move back new op lower bound from 7315833543826771968 to 7315833543663509504. Current Snapshot: MvccSnapshot[applied={T|T < 7315833543826771968}]
I20260807 08:42:53.789399 5356 tablet_replica.cc:439] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fac85f2650f94e92a772470ffc4e701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } } }
I20260807 08:42:53.789503 5356 tablet_replica.cc:442] T d30fa16ca34d404988585d4ec120a66b P d528f82452f3410a82e4e12367f0240f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 08:42:53.793923 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.790806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.28.108.129:39605)
0807 08:42:53.790854 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.790856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.790865 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:42:53.790909 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.790910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.790933 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.791018 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:53.791130 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.791904 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.792817 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.793078 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.793154 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.793477 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:53.793659 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:53.793740 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.793782 (+ 42us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.793807 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 08:42:53.800825 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.797389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:60251 (local address 127.28.108.130:36419)
0807 08:42:53.797462 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.797464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.797585 (+ 121us) server_negotiation.cc:407] Connection header received
0807 08:42:53.797691 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.797692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.797703 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.797761 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:53.797939 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.798544 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.799898 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.800260 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.800328 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.800560 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:53.800598 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:53.800655 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.800735 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.800756 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 08:42:53.801106 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.797254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:36419 (local address 127.28.108.129:60251)
0807 08:42:53.797492 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.797505 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.797587 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.797780 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.797781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.797782 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:53.797863 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.797864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.798633 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.798634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.799789 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.799793 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.800342 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.800343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.800417 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.800442 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:53.800673 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:53.800676 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.800726 (+ 50us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.800774 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":128,"threads_started":1}
I20260807 08:42:53.803797 5272 raft_consensus.cc:1261] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Refusing update from remote peer cbec3ab06bcd4f5b8191093abb0be09e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:53.803900 5113 raft_consensus.cc:1261] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Refusing update from remote peer cbec3ab06bcd4f5b8191093abb0be09e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:53.804098 5357 consensus_queue.cc:1035] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:53.804237 5382 consensus_queue.cc:1035] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:53.807180 5363 mvcc.cc:205] Tried to move back new op lower bound from 7315833543898423296 to 7315833542680715264. Current Snapshot: MvccSnapshot[applied={T|T < 7315833543898423296}]
I20260807 08:42:53.813871 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.811202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:53671 (local address 127.28.108.129:39605)
0807 08:42:53.811240 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.811242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.811386 (+ 144us) server_negotiation.cc:407] Connection header received
0807 08:42:53.811461 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.811462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.811473 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.811515 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:53.811674 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.812180 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.812970 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.813274 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.813352 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.813568 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:53.813607 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:53.813667 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.813785 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.813807 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260807 08:42:53.813911 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.811113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:39605 (local address 127.28.108.130:53671)
0807 08:42:53.811304 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.811316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.811401 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.811527 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.811528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.811529 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:53.811598 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.811599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.812265 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.812266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.812880 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.812882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.813368 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.813369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.813439 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.813463 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:53.813713 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:53.813715 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.813754 (+ 39us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.813809 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":114,"threads_started":1}
I20260807 08:42:53.825693 4791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:53.825838 4791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:53.826561 4791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b: Sending DeleteTablet for 3 replicas of tablet 1831c93381ca40c08e0c98bf7a7546a9
I20260807 08:42:53.826897 4947 tablet_service.cc:1514] Processing DeleteTablet for tablet 1831c93381ca40c08e0c98bf7a7546a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:53 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:38646
I20260807 08:42:53.827006 5373 tablet_replica.cc:330] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e: stopping tablet replica
I20260807 08:42:53.826995 5248 tablet_service.cc:1514] Processing DeleteTablet for tablet 1831c93381ca40c08e0c98bf7a7546a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:53 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:55484
I20260807 08:42:53.827050 5373 raft_consensus.cc:2229] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:53.827088 5374 tablet_replica.cc:330] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f: stopping tablet replica
I20260807 08:42:53.827137 5374 raft_consensus.cc:2229] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:53.827154 5373 raft_consensus.cc:2258] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:53.827178 5374 raft_consensus.cc:2258] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:53.827538 5374 ts_tablet_manager.cc:1905] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:53.827690 5373 ts_tablet_manager.cc:1905] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:53.829453 5089 tablet_service.cc:1514] Processing DeleteTablet for tablet 1831c93381ca40c08e0c98bf7a7546a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:53 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:33398
I20260807 08:42:53.829586 5375 tablet_replica.cc:330] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e: stopping tablet replica
I20260807 08:42:53.829667 5374 ts_tablet_manager.cc:1918] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:53.829710 5375 raft_consensus.cc:2229] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:53.829733 5374 log.cc:1196] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/1831c93381ca40c08e0c98bf7a7546a9
I20260807 08:42:53.829759 5375 raft_consensus.cc:2258] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:53.830001 5374 ts_tablet_manager.cc:1939] T 1831c93381ca40c08e0c98bf7a7546a9 P d528f82452f3410a82e4e12367f0240f: Deleting consensus metadata
I20260807 08:42:53.830411 4774 catalog_manager.cc:4841] TS d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739): tablet 1831c93381ca40c08e0c98bf7a7546a9 (table test-table [id=1fff9a0b25d74d2abac6427f301690c2]) successfully deleted
I20260807 08:42:53.833956 5373 ts_tablet_manager.cc:1918] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:53.834041 5373 log.cc:1196] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/wal/wals/1831c93381ca40c08e0c98bf7a7546a9
I20260807 08:42:53.834280 5373 ts_tablet_manager.cc:1939] T 1831c93381ca40c08e0c98bf7a7546a9 P cbec3ab06bcd4f5b8191093abb0be09e: Deleting consensus metadata
I20260807 08:42:53.834844 4776 catalog_manager.cc:4841] TS cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605): tablet 1831c93381ca40c08e0c98bf7a7546a9 (table test-table [id=1fff9a0b25d74d2abac6427f301690c2]) successfully deleted
I20260807 08:42:53.835273 5375 ts_tablet_manager.cc:1905] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:53.837592 5375 ts_tablet_manager.cc:1918] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:53.837697 5375 log.cc:1196] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/1831c93381ca40c08e0c98bf7a7546a9
I20260807 08:42:53.837941 5375 ts_tablet_manager.cc:1939] T 1831c93381ca40c08e0c98bf7a7546a9 P fac85f2650f94e92a772470ffc4e701e: Deleting consensus metadata
I20260807 08:42:53.840420 4776 catalog_manager.cc:4841] TS fac85f2650f94e92a772470ffc4e701e (127.28.108.130:36419): tablet 1831c93381ca40c08e0c98bf7a7546a9 (table test-table [id=1fff9a0b25d74d2abac6427f301690c2]) successfully deleted
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 08:42:53.858637 4849 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:53.852558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.28.108.190:42087)
0807 08:42:53.852620 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.852623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.852809 (+ 186us) server_negotiation.cc:407] Connection header received
0807 08:42:53.852901 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.852902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.852914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.852987 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:53.853633 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.854172 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.854938 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.855166 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.855245 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.858572 (+ 3327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50776: BlockingRecv error: recv got EOF from 127.0.0.1:50776 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 07 08:42:53 dist-test-slave-9zdj krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM
I20260807 08:42:53.869124 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.859937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.28.108.190:42087)
0807 08:42:53.859984 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.859986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.860219 (+ 233us) server_negotiation.cc:407] Connection header received
0807 08:42:53.860291 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.860293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.860306 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.860375 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:53.860883 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.861388 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.862187 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.862388 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.862477 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.865555 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:53.865557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:53.865558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:53.865569 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:53.867592 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.868077 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.868078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.868079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.868160 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:53.868443 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:53.868444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:53.868444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:53.868603 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:53.868670 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.868896 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.869001 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 08:42:53.871289 4791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:42:53.871512 4791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:53.873121 4947 tablet_service.cc:1467] Processing CreateTablet for tablet 586cc1f9f3654a48badb5f28f3b4d3af (DEFAULT_TABLE table=test-table [id=fb6a9672c0304408a48c9d61a15e1474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.873121 5089 tablet_service.cc:1467] Processing CreateTablet for tablet 586cc1f9f3654a48badb5f28f3b4d3af (DEFAULT_TABLE table=test-table [id=fb6a9672c0304408a48c9d61a15e1474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.873297 5089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586cc1f9f3654a48badb5f28f3b4d3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.873303 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586cc1f9f3654a48badb5f28f3b4d3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.873734 5248 tablet_service.cc:1467] Processing CreateTablet for tablet 586cc1f9f3654a48badb5f28f3b4d3af (DEFAULT_TABLE table=test-table [id=fb6a9672c0304408a48c9d61a15e1474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.873875 5248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586cc1f9f3654a48badb5f28f3b4d3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.875098 5354 tablet_bootstrap.cc:492] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e: Bootstrap starting.
I20260807 08:42:53.875458 5354 tablet_bootstrap.cc:654] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.875463 5350 tablet_bootstrap.cc:492] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f: Bootstrap starting.
I20260807 08:42:53.875901 5350 tablet_bootstrap.cc:654] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.876963 5350 tablet_bootstrap.cc:492] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f: No bootstrap required, opened a new log
I20260807 08:42:53.877023 5350 ts_tablet_manager.cc:1397] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 08:42:53.877187 5350 raft_consensus.cc:348] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.877283 5350 raft_consensus.cc:374] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.877308 5350 raft_consensus.cc:729] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d528f82452f3410a82e4e12367f0240f, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.877362 5350 consensus_queue.cc:260] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.877457 5350 ts_tablet_manager.cc:1428] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:53.877974 5351 tablet_bootstrap.cc:492] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e: Bootstrap starting.
I20260807 08:42:53.878482 5351 tablet_bootstrap.cc:654] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.878765 5356 raft_consensus.cc:482] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:53.878830 5356 raft_consensus.cc:504] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.879249 5356 leader_election.cc:288] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605), fac85f2650f94e92a772470ffc4e701e (127.28.108.130:36419)
I20260807 08:42:53.880095 5351 tablet_bootstrap.cc:492] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e: No bootstrap required, opened a new log
I20260807 08:42:53.880156 5351 ts_tablet_manager.cc:1397] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 08:42:53.880295 5354 tablet_bootstrap.cc:492] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e: No bootstrap required, opened a new log
I20260807 08:42:53.880348 5354 ts_tablet_manager.cc:1397] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260807 08:42:53.880378 5351 raft_consensus.cc:348] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.880457 5351 raft_consensus.cc:374] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.880479 5351 raft_consensus.cc:729] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbec3ab06bcd4f5b8191093abb0be09e, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.880528 5351 consensus_queue.cc:260] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.880496 5354 raft_consensus.cc:348] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.880643 5354 raft_consensus.cc:374] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.880651 5351 ts_tablet_manager.cc:1428] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 08:42:53.880677 5354 raft_consensus.cc:729] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac85f2650f94e92a772470ffc4e701e, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.880733 5354 consensus_queue.cc:260] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.880842 5354 ts_tablet_manager.cc:1428] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 08:42:53.885411 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.881028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:50333 (local address 127.28.108.129:39605)
0807 08:42:53.881166 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.881169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.882150 (+ 981us) server_negotiation.cc:407] Connection header received
0807 08:42:53.882274 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.882276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.882287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.882322 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:53.882607 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.883200 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.884377 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.884984 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.885076 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.885091 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.885273 (+ 182us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.885318 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260807 08:42:53.885967 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.879666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:39605 (local address 127.28.108.131:50333)
0807 08:42:53.882071 (+ 2405us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.882085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.882187 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.882404 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.882405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.882408 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:53.882500 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.882503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.883312 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.883313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.884253 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.884257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.885091 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.885093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.885192 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.885196 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.885267 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.885318 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2383,"thread_start_us":65,"threads_started":1}
I20260807 08:42:53.886423 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586cc1f9f3654a48badb5f28f3b4d3af" candidate_uuid: "d528f82452f3410a82e4e12367f0240f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" is_pre_election: true
I20260807 08:42:53.886528 4967 raft_consensus.cc:2454] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d528f82452f3410a82e4e12367f0240f in term 0.
I20260807 08:42:53.886833 5197 leader_election.cc:302] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbec3ab06bcd4f5b8191093abb0be09e, d528f82452f3410a82e4e12367f0240f; no voters:
I20260807 08:42:53.886996 5356 raft_consensus.cc:2781] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:53.887027 5356 raft_consensus.cc:482] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:53.887053 5356 raft_consensus.cc:3037] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.888273 5356 raft_consensus.cc:504] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.888415 5356 leader_election.cc:288] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [CANDIDATE]: Term 1 election: Requested vote from peers cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605), fac85f2650f94e92a772470ffc4e701e (127.28.108.130:36419)
I20260807 08:42:53.889160 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586cc1f9f3654a48badb5f28f3b4d3af" candidate_uuid: "d528f82452f3410a82e4e12367f0240f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbec3ab06bcd4f5b8191093abb0be09e"
I20260807 08:42:53.889242 4967 raft_consensus.cc:3037] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.889875 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.881121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:42141 (local address 127.28.108.130:36419)
0807 08:42:53.881398 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.881401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.885825 (+ 4424us) server_negotiation.cc:407] Connection header received
0807 08:42:53.885950 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.885952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.885965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.886000 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:42:53.886252 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.887035 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.888755 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.889411 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.889462 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.889474 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.889758 (+ 284us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.889806 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260807 08:42:53.890182 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.879877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:36419 (local address 127.28.108.131:42141)
0807 08:42:53.885742 (+ 5865us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:53.885756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:53.885857 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:53.886080 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:53.886081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:53.886084 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:42:53.886152 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.886154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.887201 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.887202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.888647 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:53.888650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.889560 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.889561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:53.889665 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.889670 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:42:53.889996 (+ 326us) client_negotiation.cc:240] Negotiation successful
0807 08:42:53.890047 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5847,"thread_start_us":60,"threads_started":1}
I20260807 08:42:53.890230 4967 raft_consensus.cc:2454] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d528f82452f3410a82e4e12367f0240f in term 1.
I20260807 08:42:53.890620 5197 leader_election.cc:302] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbec3ab06bcd4f5b8191093abb0be09e, d528f82452f3410a82e4e12367f0240f; no voters:
I20260807 08:42:53.890811 5356 raft_consensus.cc:2781] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:53.890944 5356 raft_consensus.cc:686] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 1 LEADER]: Becoming Leader. State: Replica: d528f82452f3410a82e4e12367f0240f, State: Running, Role: LEADER
I20260807 08:42:53.890902 5113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586cc1f9f3654a48badb5f28f3b4d3af" candidate_uuid: "d528f82452f3410a82e4e12367f0240f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac85f2650f94e92a772470ffc4e701e" is_pre_election: true
I20260807 08:42:53.891004 5113 raft_consensus.cc:2454] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d528f82452f3410a82e4e12367f0240f in term 0.
I20260807 08:42:53.891032 5356 consensus_queue.cc:237] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.891402 5112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586cc1f9f3654a48badb5f28f3b4d3af" candidate_uuid: "d528f82452f3410a82e4e12367f0240f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac85f2650f94e92a772470ffc4e701e"
I20260807 08:42:53.891505 5112 raft_consensus.cc:3037] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:53.892196 5112 raft_consensus.cc:2454] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d528f82452f3410a82e4e12367f0240f in term 1.
I20260807 08:42:53.892730 4791 catalog_manager.cc:5495] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f reported cstate change: term changed from 0 to 1, leader changed from <none> to d528f82452f3410a82e4e12367f0240f (127.28.108.131). New cstate: current_term: 1 leader_uuid: "d528f82452f3410a82e4e12367f0240f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } health_report { overall_health: HEALTHY } } }
I20260807 08:42:53.903610 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:53.900519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.28.108.131:36739)
0807 08:42:53.900704 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:53.900706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:53.900712 (+ 6us) server_negotiation.cc:407] Connection header received
0807 08:42:53.900759 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:53.900759 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:53.900783 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:53.900868 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:53.901023 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.901606 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.902648 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:53.902853 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:53.902930 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:53.903109 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:53.903325 (+ 216us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:53.903413 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:53.903478 (+ 65us) server_negotiation.cc:299] Negotiation successful
0807 08:42:53.903504 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260807 08:42:53.904757 4967 raft_consensus.cc:1261] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Refusing update from remote peer d528f82452f3410a82e4e12367f0240f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:53.904958 5112 raft_consensus.cc:1261] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Refusing update from remote peer d528f82452f3410a82e4e12367f0240f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:53.905301 5429 consensus_queue.cc:1035] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:53.905514 5356 consensus_queue.cc:1035] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:53.910239 4792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:53.910370 4792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:53.911252 4792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b: Sending DeleteTablet for 3 replicas of tablet 586cc1f9f3654a48badb5f28f3b4d3af
I20260807 08:42:53.911752 4947 tablet_service.cc:1514] Processing DeleteTablet for tablet 586cc1f9f3654a48badb5f28f3b4d3af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:53 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:38646
I20260807 08:42:53.911876 5089 tablet_service.cc:1514] Processing DeleteTablet for tablet 586cc1f9f3654a48badb5f28f3b4d3af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:53 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:33398
I20260807 08:42:53.911975 5373 tablet_replica.cc:330] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e: stopping tablet replica
I20260807 08:42:53.912046 5373 raft_consensus.cc:2229] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:53.912081 5373 raft_consensus.cc:2258] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:53.912187 5375 tablet_replica.cc:330] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e: stopping tablet replica
I20260807 08:42:53.912256 5375 raft_consensus.cc:2229] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:53.912297 5375 raft_consensus.cc:2258] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:53.912998 5248 tablet_service.cc:1514] Processing DeleteTablet for tablet 586cc1f9f3654a48badb5f28f3b4d3af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:53 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:55484
I20260807 08:42:53.913105 5374 tablet_replica.cc:330] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f: stopping tablet replica
I20260807 08:42:53.913023 4795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 08:42:53.913149 5374 raft_consensus.cc:2229] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 1 LEADER]: Raft consensus shutting down.
W20260807 08:42:53.913199 4795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:42:53.913252 5374 raft_consensus.cc:2258] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:53.913702 5374 ts_tablet_manager.cc:1905] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:53.915979 5089 tablet_service.cc:1467] Processing CreateTablet for tablet 93008d15b24f4196b7fc20fed32618ed (DEFAULT_TABLE table=test-table [id=ea46340b59bf4061b10fad3f8ac1ec9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.915979 4947 tablet_service.cc:1467] Processing CreateTablet for tablet 93008d15b24f4196b7fc20fed32618ed (DEFAULT_TABLE table=test-table [id=ea46340b59bf4061b10fad3f8ac1ec9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.916117 5089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93008d15b24f4196b7fc20fed32618ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.916119 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93008d15b24f4196b7fc20fed32618ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.917587 5351 tablet_bootstrap.cc:492] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e: Bootstrap starting.
I20260807 08:42:53.917701 5354 tablet_bootstrap.cc:492] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e: Bootstrap starting.
I20260807 08:42:53.918040 5354 tablet_bootstrap.cc:654] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.918040 5351 tablet_bootstrap.cc:654] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.918081 5374 ts_tablet_manager.cc:1918] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:53.918162 5374 log.cc:1196] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/586cc1f9f3654a48badb5f28f3b4d3af
I20260807 08:42:53.918409 5374 ts_tablet_manager.cc:1939] T 586cc1f9f3654a48badb5f28f3b4d3af P d528f82452f3410a82e4e12367f0240f: Deleting consensus metadata
I20260807 08:42:53.918560 5248 tablet_service.cc:1467] Processing CreateTablet for tablet 93008d15b24f4196b7fc20fed32618ed (DEFAULT_TABLE table=test-table [id=ea46340b59bf4061b10fad3f8ac1ec9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:42:53.918686 5248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93008d15b24f4196b7fc20fed32618ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:53.918969 5354 tablet_bootstrap.cc:492] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e: No bootstrap required, opened a new log
I20260807 08:42:53.919028 5354 ts_tablet_manager.cc:1397] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:53.919142 5354 raft_consensus.cc:348] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.919224 5354 raft_consensus.cc:374] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.919258 5354 raft_consensus.cc:729] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac85f2650f94e92a772470ffc4e701e, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.919312 5354 consensus_queue.cc:260] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.919420 5354 ts_tablet_manager.cc:1428] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 08:42:53.920058 5350 tablet_bootstrap.cc:492] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f: Bootstrap starting.
I20260807 08:42:53.920203 5375 ts_tablet_manager.cc:1905] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:53.920506 5350 tablet_bootstrap.cc:654] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:53.921604 5373 ts_tablet_manager.cc:1905] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:53.921864 4774 catalog_manager.cc:4841] TS d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739): tablet 586cc1f9f3654a48badb5f28f3b4d3af (table test-table [id=fb6a9672c0304408a48c9d61a15e1474]) successfully deleted
I20260807 08:42:53.923328 5373 ts_tablet_manager.cc:1918] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:53.923348 5350 tablet_bootstrap.cc:492] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f: No bootstrap required, opened a new log
I20260807 08:42:53.923409 5350 ts_tablet_manager.cc:1397] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260807 08:42:53.923435 5373 log.cc:1196] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-0/wal/wals/586cc1f9f3654a48badb5f28f3b4d3af
I20260807 08:42:53.923673 5373 ts_tablet_manager.cc:1939] T 586cc1f9f3654a48badb5f28f3b4d3af P cbec3ab06bcd4f5b8191093abb0be09e: Deleting consensus metadata
I20260807 08:42:53.923656 5350 raft_consensus.cc:348] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.923765 5350 raft_consensus.cc:374] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.923802 5350 raft_consensus.cc:729] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d528f82452f3410a82e4e12367f0240f, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.923856 5350 consensus_queue.cc:260] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.923957 5350 ts_tablet_manager.cc:1428] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 08:42:53.924544 5351 tablet_bootstrap.cc:492] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e: No bootstrap required, opened a new log
I20260807 08:42:53.924600 5351 ts_tablet_manager.cc:1397] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260807 08:42:53.924768 5351 raft_consensus.cc:348] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.924868 5351 raft_consensus.cc:374] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:53.924899 5351 raft_consensus.cc:729] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbec3ab06bcd4f5b8191093abb0be09e, State: Initialized, Role: FOLLOWER
I20260807 08:42:53.924947 5351 consensus_queue.cc:260] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:53.925046 5351 ts_tablet_manager.cc:1428] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 08:42:53.925911 5375 ts_tablet_manager.cc:1918] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:42:53.926036 5375 log.cc:1196] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/586cc1f9f3654a48badb5f28f3b4d3af
I20260807 08:42:53.926290 5375 ts_tablet_manager.cc:1939] T 586cc1f9f3654a48badb5f28f3b4d3af P fac85f2650f94e92a772470ffc4e701e: Deleting consensus metadata
I20260807 08:42:53.926753 4776 catalog_manager.cc:4841] TS fac85f2650f94e92a772470ffc4e701e (127.28.108.130:36419): tablet 586cc1f9f3654a48badb5f28f3b4d3af (table test-table [id=fb6a9672c0304408a48c9d61a15e1474]) successfully deleted
I20260807 08:42:53.927194 4776 catalog_manager.cc:4841] TS cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605): tablet 586cc1f9f3654a48badb5f28f3b4d3af (table test-table [id=fb6a9672c0304408a48c9d61a15e1474]) successfully deleted
I20260807 08:42:54.003873 5368 raft_consensus.cc:482] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:42:54.004009 5368 raft_consensus.cc:504] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:54.004194 5368 leader_election.cc:288] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605), d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739)
I20260807 08:42:54.004495 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93008d15b24f4196b7fc20fed32618ed" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" is_pre_election: true
I20260807 08:42:54.004478 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93008d15b24f4196b7fc20fed32618ed" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d528f82452f3410a82e4e12367f0240f" is_pre_election: true
I20260807 08:42:54.004608 5272 raft_consensus.cc:2454] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 0.
I20260807 08:42:54.004607 4967 raft_consensus.cc:2454] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 0.
I20260807 08:42:54.004791 5042 leader_election.cc:302] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbec3ab06bcd4f5b8191093abb0be09e, fac85f2650f94e92a772470ffc4e701e; no voters:
I20260807 08:42:54.004907 5368 raft_consensus.cc:2781] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:42:54.004952 5368 raft_consensus.cc:482] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:42:54.004976 5368 raft_consensus.cc:3037] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:54.005995 5368 raft_consensus.cc:504] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:54.006173 5368 leader_election.cc:288] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 election: Requested vote from peers cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605), d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739)
I20260807 08:42:54.006383 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93008d15b24f4196b7fc20fed32618ed" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbec3ab06bcd4f5b8191093abb0be09e"
I20260807 08:42:54.006496 4967 raft_consensus.cc:3037] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:54.006448 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93008d15b24f4196b7fc20fed32618ed" candidate_uuid: "fac85f2650f94e92a772470ffc4e701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d528f82452f3410a82e4e12367f0240f"
I20260807 08:42:54.006546 5272 raft_consensus.cc:3037] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:54.007218 4967 raft_consensus.cc:2454] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 1.
I20260807 08:42:54.007472 5042 leader_election.cc:302] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbec3ab06bcd4f5b8191093abb0be09e, fac85f2650f94e92a772470ffc4e701e; no voters:
I20260807 08:42:54.007512 5272 raft_consensus.cc:2454] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac85f2650f94e92a772470ffc4e701e in term 1.
I20260807 08:42:54.007642 5368 raft_consensus.cc:2781] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:54.007721 5368 raft_consensus.cc:686] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 1 LEADER]: Becoming Leader. State: Replica: fac85f2650f94e92a772470ffc4e701e, State: Running, Role: LEADER
I20260807 08:42:54.007792 5368 consensus_queue.cc:237] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } }
I20260807 08:42:54.008581 4795 catalog_manager.cc:5495] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e reported cstate change: term changed from 0 to 1, leader changed from <none> to fac85f2650f94e92a772470ffc4e701e (127.28.108.130). New cstate: current_term: 1 leader_uuid: "fac85f2650f94e92a772470ffc4e701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac85f2650f94e92a772470ffc4e701e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 } health_report { overall_health: UNKNOWN } } }
I20260807 08:42:54.025795 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:54.022382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.28.108.130:36419)
0807 08:42:54.022453 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:54.022456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:54.022571 (+ 115us) server_negotiation.cc:407] Connection header received
0807 08:42:54.022657 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:54.022658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:54.022678 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:54.022755 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:54.023014 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:54.023800 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:54.024810 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:54.025146 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:54.025228 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:54.025461 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:54.025505 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:54.025560 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:54.025640 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 08:42:54.025671 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 08:42:54.031343 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:54.028751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:52331 (local address 127.28.108.130:36419)
0807 08:42:54.028885 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:54.028887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:54.028897 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:54.028922 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:54.028923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:54.028936 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:54.029101 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:42:54.029350 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:54.029904 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:54.030658 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:54.030842 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:54.030922 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:54.031061 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:42:54.031097 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:42:54.031152 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:54.031216 (+ 64us) server_negotiation.cc:299] Negotiation successful
0807 08:42:54.031242 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":110}
I20260807 08:42:54.031343 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:54.028530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:36419 (local address 127.28.108.130:52331)
0807 08:42:54.028597 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:54.028610 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:54.028656 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:54.029108 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:54.029109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:54.029110 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 08:42:54.029176 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:54.029177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:54.030002 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:54.030003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:54.030560 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:54.030562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:54.030921 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:54.030922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:54.030976 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:54.030984 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 08:42:54.031153 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 08:42:54.031155 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 08:42:54.031209 (+ 54us) client_negotiation.cc:240] Negotiation successful
0807 08:42:54.031270 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":3}
I20260807 08:42:54.034106 5272 raft_consensus.cc:1261] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Refusing update from remote peer fac85f2650f94e92a772470ffc4e701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:54.034241 4967 raft_consensus.cc:1261] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Refusing update from remote peer fac85f2650f94e92a772470ffc4e701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:42:54.034554 5368 consensus_queue.cc:1035] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d528f82452f3410a82e4e12367f0240f" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:54.034723 5360 consensus_queue.cc:1035] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbec3ab06bcd4f5b8191093abb0be09e" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 39605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:42:54.047551 4792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:54.047690 4792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:42:54.048491 4792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28f98d824b3a416389243999efeb569b: Sending DeleteTablet for 3 replicas of tablet 93008d15b24f4196b7fc20fed32618ed
I20260807 08:42:54.048873 4947 tablet_service.cc:1514] Processing DeleteTablet for tablet 93008d15b24f4196b7fc20fed32618ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:54 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:38646
I20260807 08:42:54.049034 5373 tablet_replica.cc:330] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e: stopping tablet replica
I20260807 08:42:54.049118 5373 raft_consensus.cc:2229] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:54.049176 5373 raft_consensus.cc:2258] T 93008d15b24f4196b7fc20fed32618ed P cbec3ab06bcd4f5b8191093abb0be09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:54.049374 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 4873
I20260807 08:42:54.054723 5248 tablet_service.cc:1514] Processing DeleteTablet for tablet 93008d15b24f4196b7fc20fed32618ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:54 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:55484
I20260807 08:42:54.054934 5374 tablet_replica.cc:330] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f: stopping tablet replica
I20260807 08:42:54.055030 5374 raft_consensus.cc:2229] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:42:54.055084 5374 raft_consensus.cc:2258] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:54.055531 5089 tablet_service.cc:1514] Processing DeleteTablet for tablet 93008d15b24f4196b7fc20fed32618ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:42:54 UTC) from {username='oryx', principal='oryx/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:33398
I20260807 08:42:54.055673 5374 ts_tablet_manager.cc:1905] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:54.055748 5375 tablet_replica.cc:330] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e: stopping tablet replica
I20260807 08:42:54.055850 5375 raft_consensus.cc:2229] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:42:54.055950 5375 raft_consensus.cc:2258] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:42:54.056442 5375 ts_tablet_manager.cc:1905] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:42:54.057400 5374 ts_tablet_manager.cc:1918] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:54.057472 5374 log.cc:1196] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/93008d15b24f4196b7fc20fed32618ed
I20260807 08:42:54.057761 5374 ts_tablet_manager.cc:1939] T 93008d15b24f4196b7fc20fed32618ed P d528f82452f3410a82e4e12367f0240f: Deleting consensus metadata
I20260807 08:42:54.058216 4774 catalog_manager.cc:4841] TS d528f82452f3410a82e4e12367f0240f (127.28.108.131:36739): tablet 93008d15b24f4196b7fc20fed32618ed (table test-table [id=ea46340b59bf4061b10fad3f8ac1ec9e]) successfully deleted
I20260807 08:42:54.058570 5375 ts_tablet_manager.cc:1918] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 08:42:54.058634 5375 log.cc:1196] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/93008d15b24f4196b7fc20fed32618ed
I20260807 08:42:54.058893 5375 ts_tablet_manager.cc:1939] T 93008d15b24f4196b7fc20fed32618ed P fac85f2650f94e92a772470ffc4e701e: Deleting consensus metadata
I20260807 08:42:54.059604 4776 catalog_manager.cc:4841] TS fac85f2650f94e92a772470ffc4e701e (127.28.108.130:36419): tablet 93008d15b24f4196b7fc20fed32618ed (table test-table [id=ea46340b59bf4061b10fad3f8ac1ec9e]) successfully deleted
W20260807 08:42:54.063359 4776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:39605 (error 108)
I20260807 08:42:54.063540 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 5023
I20260807 08:42:54.065889 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:54.063830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:39605 (local address 127.0.0.1:49674)
0807 08:42:54.065745 (+ 1915us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:54.065828 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:39605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1889,"thread_start_us":41,"threads_started":1}
W20260807 08:42:54.066027 4776 catalog_manager.cc:4584] TS cbec3ab06bcd4f5b8191093abb0be09e (127.28.108.129:39605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 93008d15b24f4196b7fc20fed32618ed: Network error: Client connection negotiation failed: client connection to 127.28.108.129:39605: connect: Connection refused (error 111)
I20260807 08:42:54.075565 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 5175
I20260807 08:42:54.085601 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 4757
2026-08-07T08:42:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2867 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:54 dist-test-slave-9zdj krb5kdc[5449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:54 dist-test-slave-9zdj krb5kdc[5449](info): set up 2 sockets
Aug 07 08:42:54 dist-test-slave-9zdj krb5kdc[5449](info): commencing operation
krb5kdc: starting...
W20260807 08:42:55.220247 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:55 dist-test-slave-9zdj krb5kdc[5449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:55.370882 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:44795
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:34735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:44795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:55.460898 5463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:55.461198 5463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:55.461243 5463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:55.462538 5463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:55.462630 5463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:55.462658 5463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:55.462684 5463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:55.462713 5463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:55.464432 5463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:44795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:44795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:55.464797 5463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:55.465067 5463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:55.471302 5472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:55.471431 5468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:55.471371 5469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:55.471524 5463 server_base.cc:1034] running on GCE node
I20260807 08:42:55.471917 5463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:55.472240 5463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:55.473414 5463 hybrid_clock.cc:648] HybridClock initialized: now 1786092175473382 us; error 64 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T08:42:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1389 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:42:55 dist-test-slave-9zdj krb5kdc[5477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:42:55 dist-test-slave-9zdj krb5kdc[5477](info): set up 2 sockets
Aug 07 08:42:55 dist-test-slave-9zdj krb5kdc[5477](info): commencing operation
krb5kdc: starting...
W20260807 08:42:56.681433 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:42:56 dist-test-slave-9zdj krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:42:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:42:56.829548 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42149
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:42149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:56.927558 5491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:56.927845 5491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:56.927888 5491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:56.929183 5491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:56.929260 5491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:56.929296 5491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:56.929327 5491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:56.929353 5491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:56.931058 5491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:42149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:56.931386 5491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:56.931643 5491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:56.937368 5496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:56.937436 5491 server_base.cc:1034] running on GCE node
W20260807 08:42:56.937321 5497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:56.937561 5500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:56.937898 5491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:56.938261 5491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:56.939460 5491 hybrid_clock.cc:648] HybridClock initialized: now 1786092176939431 us; error 39 us; skew 500 ppm
Aug 07 08:42:56 dist-test-slave-9zdj krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:56.946305 5491 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:56.946835 5491 webserver.cc:460] Webserver started at http://127.28.108.190:46581/ using document root <none> and password file <none>
I20260807 08:42:56.947058 5491 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:56.947121 5491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:56.947237 5491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:56.948388 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "6853e717bb3243b38fb87920ffddc854"
format_stamp: "Formatted at 2026-08-07 08:42:56 on dist-test-slave-9zdj"
server_key: "15728b3c13f9ba42d8c30dbec6492bcd"
server_key_iv: "58cd6852173936a66775ff5a67ceb038"
server_key_version: "encryptionkey@0"
I20260807 08:42:56.948762 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "6853e717bb3243b38fb87920ffddc854"
format_stamp: "Formatted at 2026-08-07 08:42:56 on dist-test-slave-9zdj"
server_key: "15728b3c13f9ba42d8c30dbec6492bcd"
server_key_iv: "58cd6852173936a66775ff5a67ceb038"
server_key_version: "encryptionkey@0"
I20260807 08:42:56.950438 5491 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 08:42:56.951363 5507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:56.951601 5491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:56.951704 5491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "6853e717bb3243b38fb87920ffddc854"
format_stamp: "Formatted at 2026-08-07 08:42:56 on dist-test-slave-9zdj"
server_key: "15728b3c13f9ba42d8c30dbec6492bcd"
server_key_iv: "58cd6852173936a66775ff5a67ceb038"
server_key_version: "encryptionkey@0"
I20260807 08:42:56.951821 5491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:56.964891 5491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:56.967800 5491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:56.967952 5491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:56.972846 5491 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:42149
I20260807 08:42:56.972914 5569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:42149 every 8 connection(s)
I20260807 08:42:56.973470 5491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:56.974771 5570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:42:56.977214 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 5491
I20260807 08:42:56.977353 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:42:56.977587 29106 external_mini_cluster.cc:1428] Setting key 3f58a11639d39068f2e92794ec6301e7
I20260807 08:42:56.977736 5570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854: Bootstrap starting.
I20260807 08:42:56.978564 5570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854: Neither blocks nor log segments found. Creating new log.
I20260807 08:42:56.978927 5570 log.cc:824] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:56.979769 5570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854: No bootstrap required, opened a new log
I20260807 08:42:56.982118 5570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } }
I20260807 08:42:56.982326 5570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:42:56.982380 5570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6853e717bb3243b38fb87920ffddc854, State: Initialized, Role: FOLLOWER
I20260807 08:42:56.982578 5570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } }
I20260807 08:42:56.982694 5570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:56.982729 5570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:56.982784 5570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:42:56.983716 5570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } }
I20260807 08:42:56.983831 5570 leader_election.cc:302] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6853e717bb3243b38fb87920ffddc854; no voters:
I20260807 08:42:56.984165 5570 leader_election.cc:288] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:42:56.984318 5575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:42:56.984508 5570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:56.984522 5575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 1 LEADER]: Becoming Leader. State: Replica: 6853e717bb3243b38fb87920ffddc854, State: Running, Role: LEADER
I20260807 08:42:56.984755 5575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } }
I20260807 08:42:56.985157 5575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6853e717bb3243b38fb87920ffddc854. Latest consensus state: current_term: 1 leader_uuid: "6853e717bb3243b38fb87920ffddc854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } } }
I20260807 08:42:56.985427 5575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:56.985349 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6853e717bb3243b38fb87920ffddc854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } } }
I20260807 08:42:56.985976 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [sys.catalog]: This master's current role is: LEADER
Aug 07 08:42:56 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
W20260807 08:42:56.987322 5593 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:42:56.987469 5593 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:42:56.987551 5586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:56.987811 5586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:56.989864 5586 catalog_manager.cc:1347] Generated new cluster ID: 84af19a151ca47a8b17333c3075b68f3
I20260807 08:42:56.989934 5586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:56.991662 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:56.978308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.28.108.190:42149)
0807 08:42:56.978545 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:56.978558 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:56.978603 (+ 45us) server_negotiation.cc:407] Connection header received
0807 08:42:56.979242 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:56.979251 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:56.979312 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:56.979486 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:56.980361 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:56.981439 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:56.982460 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:56.983230 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:56.983376 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:56.987336 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:56.987342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:56.987342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:56.987360 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:56.989534 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:56.990055 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:56.990059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:56.990060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:56.990194 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:56.990487 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:56.990488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:56.990488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:56.990891 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:56.991080 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:56.991187 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 08:42:56.991408 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":107,"threads_started":1}
I20260807 08:42:57.033385 5586 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:42:57.034229 5586 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:57.040642 5586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854: Generated new TSK 0
I20260807 08:42:57.040921 5586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:42:57.090291 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42149
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:57.188196 5598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:57.188465 5598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:57.188498 5598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:57.189967 5598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:57.190021 5598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:57.190054 5598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:57.191785 5598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:57.192026 5598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:57.192257 5598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:57.198616 5604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:57.198625 5603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:57.198976 5598 server_base.cc:1034] running on GCE node
W20260807 08:42:57.198644 5606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:57.199528 5598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:57.199846 5598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:57.201027 5598 hybrid_clock.cc:648] HybridClock initialized: now 1786092177201022 us; error 43 us; skew 500 ppm
Aug 07 08:42:57 dist-test-slave-9zdj krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:57.204015 5598 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:42:57.204644 5598 webserver.cc:460] Webserver started at http://127.28.108.129:46459/ using document root <none> and password file <none>
I20260807 08:42:57.204926 5598 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:57.205004 5598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:57.205192 5598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:57.206589 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "372f821a35184bcb9387ea8f467f0fdf"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "b89a4c05c26a63ca71e7eaa605b79ed3"
server_key_iv: "be13d8903af19cee3dea812cab515be7"
server_key_version: "encryptionkey@0"
I20260807 08:42:57.207018 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "372f821a35184bcb9387ea8f467f0fdf"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "b89a4c05c26a63ca71e7eaa605b79ed3"
server_key_iv: "be13d8903af19cee3dea812cab515be7"
server_key_version: "encryptionkey@0"
I20260807 08:42:57.208612 5598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:42:57.209570 5614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:57.209921 5598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:57.210060 5598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "372f821a35184bcb9387ea8f467f0fdf"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "b89a4c05c26a63ca71e7eaa605b79ed3"
server_key_iv: "be13d8903af19cee3dea812cab515be7"
server_key_version: "encryptionkey@0"
I20260807 08:42:57.210155 5598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:57.222205 5598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:57.225080 5598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:57.225302 5598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:57.225720 5598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:57.226125 5598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:57.226189 5598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:57.226239 5598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:57.226269 5598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:57.233235 5598 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:37503
I20260807 08:42:57.233306 5733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:37503 every 8 connection(s)
I20260807 08:42:57.233709 5598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:42:57.238371 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 5598
I20260807 08:42:57.238535 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:42:57.238806 29106 external_mini_cluster.cc:1428] Setting key 92b0662fe84049e05bcdc08c2f9db4f9
Aug 07 08:42:57 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:57.245654 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:57.234692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:48937 (local address 127.28.108.190:42149)
0807 08:42:57.234767 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:57.234769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:57.235027 (+ 258us) server_negotiation.cc:407] Connection header received
0807 08:42:57.235630 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:57.235631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:57.235645 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:57.235708 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:57.236861 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.237444 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.238865 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.239168 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.239259 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:57.242105 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:57.242107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:57.242107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:57.242117 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:57.243862 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:57.244404 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:57.244406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:57.244406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:57.244478 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:57.244787 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:57.244789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:57.244790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:57.244939 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:57.245026 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:57.245391 (+ 365us) server_negotiation.cc:299] Negotiation successful
0807 08:42:57.245511 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 08:42:57.245890 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:57.234647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:48937)
0807 08:42:57.234868 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:57.234913 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:57.235500 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:57.235801 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:57.235805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:57.236501 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:57.236734 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:57.236740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.237574 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.237575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:57.238670 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:57.238673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.239280 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.239281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:57.239432 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:57.240042 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:57.240053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:57.241974 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:57.243990 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:57.244001 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:57.244005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:57.244311 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:57.244559 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:57.244560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:57.244560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:57.244706 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:57.245028 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:57.245037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:57.245230 (+ 193us) client_negotiation.cc:769] Sending connection context
0807 08:42:57.245351 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 08:42:57.245555 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":118,"threads_started":1}
I20260807 08:42:57.246457 5735 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42149
I20260807 08:42:57.246601 5735 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:57.246863 5735 heartbeater.cc:511] Master 127.28.108.190:42149 requested a full tablet report, sending...
I20260807 08:42:57.247714 5527 ts_manager.cc:194] Registered new tserver with Master: 372f821a35184bcb9387ea8f467f0fdf (127.28.108.129:37503)
I20260807 08:42:57.248862 5527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:48937
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:42:57.277153 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42149
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:57.365268 5741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:57.365505 5741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:57.365531 5741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:57.366763 5741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:57.366816 5741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:57.366845 5741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:57.368386 5741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:57.368633 5741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:57.368844 5741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:57.374491 5747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:57.374483 5749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:57.374483 5746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:57.374826 5741 server_base.cc:1034] running on GCE node
I20260807 08:42:57.375067 5741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:57.375339 5741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:57.376518 5741 hybrid_clock.cc:648] HybridClock initialized: now 1786092177376500 us; error 35 us; skew 500 ppm
Aug 07 08:42:57 dist-test-slave-9zdj krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:57.379712 5741 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:42:57.380373 5741 webserver.cc:460] Webserver started at http://127.28.108.130:40813/ using document root <none> and password file <none>
I20260807 08:42:57.380646 5741 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:57.380717 5741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:57.380864 5741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:57.382093 5741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "585caeb799d94e6e9e8dd332d262e410"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "bb415ee6f23477981188ffd02f0aacc3"
server_key_iv: "50e0e5c26c698323fb38e1ad6560a7ea"
server_key_version: "encryptionkey@0"
I20260807 08:42:57.382634 5741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "585caeb799d94e6e9e8dd332d262e410"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "bb415ee6f23477981188ffd02f0aacc3"
server_key_iv: "50e0e5c26c698323fb38e1ad6560a7ea"
server_key_version: "encryptionkey@0"
I20260807 08:42:57.384279 5741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:42:57.385335 5756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:57.385596 5741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:57.385725 5741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "585caeb799d94e6e9e8dd332d262e410"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "bb415ee6f23477981188ffd02f0aacc3"
server_key_iv: "50e0e5c26c698323fb38e1ad6560a7ea"
server_key_version: "encryptionkey@0"
I20260807 08:42:57.385874 5741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:57.401571 5741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:57.404453 5741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:57.404593 5741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:57.404819 5741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:57.405228 5741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:57.405273 5741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:57.405306 5741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:57.405321 5741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:57.411873 5741 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:38451
I20260807 08:42:57.411953 5875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:38451 every 8 connection(s)
I20260807 08:42:57.412333 5741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:57.414217 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 5741
I20260807 08:42:57.414475 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:42:57.414723 29106 external_mini_cluster.cc:1428] Setting key 916b74ccd81e5db23ba2d5fa052086e9
Aug 07 08:42:57 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:57.425518 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:57.413238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:54807 (local address 127.28.108.190:42149)
0807 08:42:57.413364 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:57.413367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:57.413570 (+ 203us) server_negotiation.cc:407] Connection header received
0807 08:42:57.414406 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:57.414409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:57.414426 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:57.414511 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:57.415916 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.416748 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.418426 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.418648 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.418713 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:57.421922 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:57.421923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:57.421924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:57.421934 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:57.423642 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:57.424286 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:57.424287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:57.424288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:57.424364 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:57.424684 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:57.424686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:57.424686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:57.424845 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:57.424925 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:57.425296 (+ 371us) server_negotiation.cc:299] Negotiation successful
0807 08:42:57.425409 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
I20260807 08:42:57.425861 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:57.413140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:54807)
0807 08:42:57.413401 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:57.413454 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:57.414235 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:57.414623 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:57.414628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:57.415515 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:57.415812 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:57.415818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.416920 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.416921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:57.418319 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:57.418322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.418779 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.418780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:57.418984 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:57.419635 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:57.419648 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:57.421793 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:57.423788 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:57.423800 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:57.423804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:57.424159 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:57.424442 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:57.424443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:57.424444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:57.424589 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:57.424930 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:57.424939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:57.425118 (+ 179us) client_negotiation.cc:769] Sending connection context
0807 08:42:57.425253 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 08:42:57.425468 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260807 08:42:57.426524 5876 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42149
I20260807 08:42:57.426653 5876 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:57.426944 5876 heartbeater.cc:511] Master 127.28.108.190:42149 requested a full tablet report, sending...
I20260807 08:42:57.427454 5527 ts_manager.cc:194] Registered new tserver with Master: 585caeb799d94e6e9e8dd332d262e410 (127.28.108.130:38451)
I20260807 08:42:57.427873 5527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:54807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:42:57.450589 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42149
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:57.540753 5881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:57.541018 5881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:57.541045 5881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:57.542373 5881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:57.542425 5881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:57.542454 5881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:57.543974 5881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:57.544201 5881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:57.544407 5881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:42:57.550503 5881 server_base.cc:1034] running on GCE node
W20260807 08:42:57.550526 5888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:57.550508 5889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:57.550549 5892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:57.551080 5881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:57.551354 5881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:57.552567 5881 hybrid_clock.cc:648] HybridClock initialized: now 1786092177552551 us; error 31 us; skew 500 ppm
Aug 07 08:42:57 dist-test-slave-9zdj krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:57.555832 5881 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:42:57.556468 5881 webserver.cc:460] Webserver started at http://127.28.108.131:34891/ using document root <none> and password file <none>
I20260807 08:42:57.556774 5881 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:57.556849 5881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:57.557019 5881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:42:57.558308 5881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "30b2b91aad984dbabb2d41fcaa4a11b0"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "ac28df650407e3aead8d6c082c6f1f8e"
server_key_iv: "1b06e8e66c678c8e2965ad5d146cd0be"
server_key_version: "encryptionkey@0"
I20260807 08:42:57.558723 5881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "30b2b91aad984dbabb2d41fcaa4a11b0"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "ac28df650407e3aead8d6c082c6f1f8e"
server_key_iv: "1b06e8e66c678c8e2965ad5d146cd0be"
server_key_version: "encryptionkey@0"
I20260807 08:42:57.560441 5881 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 08:42:57.561602 5899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:57.561985 5881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:57.562220 5881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "30b2b91aad984dbabb2d41fcaa4a11b0"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "ac28df650407e3aead8d6c082c6f1f8e"
server_key_iv: "1b06e8e66c678c8e2965ad5d146cd0be"
server_key_version: "encryptionkey@0"
I20260807 08:42:57.562319 5881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:57.571822 5881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:57.575022 5881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:57.575253 5881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:57.575526 5881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:57.575906 5881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:57.575963 5881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:57.576009 5881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:57.576035 5881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:57.582639 5881 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:46333
I20260807 08:42:57.582696 6015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:46333 every 8 connection(s)
I20260807 08:42:57.583091 5881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:57.587333 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 5881
I20260807 08:42:57.587462 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:42:57.587652 29106 external_mini_cluster.cc:1428] Setting key 8602f54f2e2dc98487a74622064535a4
Aug 07 08:42:57 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:42:57.595350 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:57.584055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:41765 (local address 127.28.108.190:42149)
0807 08:42:57.584122 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:57.584125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:57.584373 (+ 248us) server_negotiation.cc:407] Connection header received
0807 08:42:57.584925 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:57.584927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:57.584943 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:57.585033 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:57.586386 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.586987 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.588746 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.589029 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.589126 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:57.591596 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:57.591599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:57.591599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:57.591607 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:57.593532 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:57.594082 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:57.594083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:57.594084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:57.594149 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:57.594496 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:57.594497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:57.594498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:57.594649 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:57.594704 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:57.595109 (+ 405us) server_negotiation.cc:299] Negotiation successful
0807 08:42:57.595191 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 08:42:57.595422 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:57.583901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:41765)
0807 08:42:57.584093 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:57.584137 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:57.584763 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:57.585177 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:57.585181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:57.586004 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:57.586233 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:57.586239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.587114 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.587115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:57.588637 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:57.588640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:57.589128 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:57.589129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:57.589282 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:57.589923 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:57.589933 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:57.591488 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:42:57.593710 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:57.593722 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:57.593724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:57.593999 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:57.594242 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:42:57.594243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:42:57.594244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:42:57.594413 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:42:57.594746 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:42:57.594755 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:42:57.594934 (+ 179us) client_negotiation.cc:769] Sending connection context
0807 08:42:57.595044 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 08:42:57.595218 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260807 08:42:57.596181 6016 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42149
I20260807 08:42:57.596302 6016 heartbeater.cc:464] Registering TS with master...
I20260807 08:42:57.596488 6016 heartbeater.cc:511] Master 127.28.108.190:42149 requested a full tablet report, sending...
I20260807 08:42:57.596943 5527 ts_manager.cc:194] Registered new tserver with Master: 30b2b91aad984dbabb2d41fcaa4a11b0 (127.28.108.131:46333)
I20260807 08:42:57.597419 5527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:41765
I20260807 08:42:57.599992 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab.
Entry for principal oryx/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab.
WARNING: no policy specified for oryx/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab.
Entry for principal oryx/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab.
WARNING: no policy specified for oryx/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab.
Entry for principal oryx/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab.
I20260807 08:42:57.698838 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 5598
I20260807 08:42:57.706498 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 5741
I20260807 08:42:57.713914 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 5881
I20260807 08:42:57.722137 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 5491
I20260807 08:42:57.731688 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42149
--webserver_interface=127.28.108.190
--webserver_port=46581
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:42149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:42:57.843503 6025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:57.843777 6025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:57.843806 6025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:57.845090 6025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:42:57.845161 6025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:57.845177 6025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:57.845191 6025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:42:57.845204 6025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:42:57.846879 6025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:42149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=46581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:57.847146 6025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:57.847366 6025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:42:57.854293 6025 server_base.cc:1034] running on GCE node
W20260807 08:42:57.854271 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:57.854264 6036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:57.854271 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:57.855248 6025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:57.855895 6025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:57.857126 6025 hybrid_clock.cc:648] HybridClock initialized: now 1786092177857118 us; error 50 us; skew 500 ppm
Aug 07 08:42:57 dist-test-slave-9zdj krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:57.861120 6025 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:42:57.861857 6025 webserver.cc:460] Webserver started at http://127.28.108.190:46581/ using document root <none> and password file <none>
I20260807 08:42:57.862145 6025 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:57.862250 6025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:57.864248 6025 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:42:57.865458 6043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:57.865895 6025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:57.866044 6025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "6853e717bb3243b38fb87920ffddc854"
format_stamp: "Formatted at 2026-08-07 08:42:56 on dist-test-slave-9zdj"
server_key: "15728b3c13f9ba42d8c30dbec6492bcd"
server_key_iv: "58cd6852173936a66775ff5a67ceb038"
server_key_version: "encryptionkey@0"
I20260807 08:42:57.866547 6025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:57.884014 6025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:57.887483 6025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:57.887728 6025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:57.893110 6025 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:42149
I20260807 08:42:57.893342 6100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:42149 every 8 connection(s)
I20260807 08:42:57.893721 6025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:42:57.895493 6101 sys_catalog.cc:263] Verifying existing consensus state
I20260807 08:42:57.896260 6101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854: Bootstrap starting.
I20260807 08:42:57.899360 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 6025
I20260807 08:42:57.899760 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:37503
--local_ip_for_outbound_sockets=127.28.108.129
--tserver_master_addrs=127.28.108.190:42149
--webserver_port=46459
--webserver_interface=127.28.108.129
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.108.129
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:42:57.906575 6101 log.cc:824] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854: Log is configured to *not* fsync() on all Append() calls
I20260807 08:42:57.908738 6101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:42:57.909091 6101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854: Bootstrap complete.
I20260807 08:42:57.911777 6101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } }
I20260807 08:42:57.912032 6101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6853e717bb3243b38fb87920ffddc854, State: Initialized, Role: FOLLOWER
I20260807 08:42:57.912216 6101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } }
I20260807 08:42:57.912300 6101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:42:57.912364 6101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:42:57.912413 6101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:42:57.913558 6101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } }
I20260807 08:42:57.913719 6101 leader_election.cc:302] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6853e717bb3243b38fb87920ffddc854; no voters:
I20260807 08:42:57.913944 6101 leader_election.cc:288] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 08:42:57.913986 6106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:42:57.914155 6106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [term 2 LEADER]: Becoming Leader. State: Replica: 6853e717bb3243b38fb87920ffddc854, State: Running, Role: LEADER
I20260807 08:42:57.914266 6101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:42:57.914322 6106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } }
I20260807 08:42:57.914727 6108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6853e717bb3243b38fb87920ffddc854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } } }
I20260807 08:42:57.914837 6108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:57.915302 6109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6853e717bb3243b38fb87920ffddc854. Latest consensus state: current_term: 2 leader_uuid: "6853e717bb3243b38fb87920ffddc854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6853e717bb3243b38fb87920ffddc854" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42149 } } }
I20260807 08:42:57.915375 6109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854 [sys.catalog]: This master's current role is: LEADER
I20260807 08:42:57.915540 6113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:42:57.916136 6113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:42:57.916273 6113 catalog_manager.cc:1259] Loaded cluster ID: 84af19a151ca47a8b17333c3075b68f3
I20260807 08:42:57.916368 6113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:42:57.918015 6113 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:42:57.918236 6113 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6853e717bb3243b38fb87920ffddc854: Loaded TSK: 0
I20260807 08:42:57.918522 6113 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 08:42:57.995689 6103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:57.995965 6103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:57.996013 6103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:57.997323 6103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:57.997401 6103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:57.997457 6103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:42:57.999123 6103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.129.keytab
--principal=oryx/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:37503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=46459
--tserver_master_addrs=127.28.108.190:42149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:57.999440 6103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:57.999717 6103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:58.007076 6130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:58.007086 6129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:58.007109 6133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:58.007511 6103 server_base.cc:1034] running on GCE node
I20260807 08:42:58.007941 6103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:58.008270 6103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:58.009459 6103 hybrid_clock.cc:648] HybridClock initialized: now 1786092178009434 us; error 45 us; skew 500 ppm
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092178, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:58.012866 6103 init.cc:377] Logged in from keytab as oryx/127.28.108.129@KRBTEST.COM (short username oryx)
I20260807 08:42:58.013747 6103 webserver.cc:460] Webserver started at http://127.28.108.129:46459/ using document root <none> and password file <none>
I20260807 08:42:58.014138 6103 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:58.014286 6103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:58.016244 6103 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:42:58.017031 6142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:58.017366 6103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:58.017496 6103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "372f821a35184bcb9387ea8f467f0fdf"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "b89a4c05c26a63ca71e7eaa605b79ed3"
server_key_iv: "be13d8903af19cee3dea812cab515be7"
server_key_version: "encryptionkey@0"
I20260807 08:42:58.017998 6103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:58.030197 6103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:58.033600 6103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:58.033947 6103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:58.034214 6103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:58.034585 6103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:58.034649 6103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:58.034696 6103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:58.034723 6103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:58.040946 6103 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:37503
I20260807 08:42:58.041102 6263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:37503 every 8 connection(s)
I20260807 08:42:58.041446 6103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:42:58.047662 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 6103
I20260807 08:42:58.048151 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:38451
--local_ip_for_outbound_sockets=127.28.108.130
--tserver_master_addrs=127.28.108.190:42149
--webserver_port=40813
--webserver_interface=127.28.108.130
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.108.130
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:42:58.053148 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:58.042285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:59823)
0807 08:42:58.042507 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:58.042560 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:58.043217 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:58.043850 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:58.043855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:58.044683 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:58.044999 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.045007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.046559 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.046560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.047913 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.047917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.049095 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.049097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.049261 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.050017 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:58.050034 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:58.052866 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":101,"threads_started":1}
W20260807 08:42:58.053570 6264 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:42149 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.108.190:42149: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
W20260807 08:42:58.054265 6267 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:58.042413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:59823 (local address 127.28.108.190:42149)
0807 08:42:58.042643 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:58.042651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:58.042681 (+ 30us) server_negotiation.cc:407] Connection header received
0807 08:42:58.043555 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:58.043562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:58.043623 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:58.043753 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:58.045187 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.046436 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.048012 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.048832 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.048963 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.053128 (+ 4165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:59823: BlockingRecv error: recv got EOF from 127.28.108.129:59823 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:42:58.060834 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:58.053874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:44789)
0807 08:42:58.053924 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:58.053939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:58.053996 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:58.054595 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:58.054597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:58.054782 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:58.054862 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.054865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.055978 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.055980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.056890 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.056896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.057523 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.057555 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.057711 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.058318 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:58.058325 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:58.060733 (+ 2408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260807 08:42:58.060993 6269 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:58.053987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:44789 (local address 127.28.108.190:42149)
0807 08:42:58.054194 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:58.054196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:58.054205 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:42:58.054236 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:58.054236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:58.054250 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:58.054563 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:58.054966 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.055867 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.057017 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.057411 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.057505 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.060894 (+ 3389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:44789: BlockingRecv error: recv got EOF from 127.28.108.129:44789 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":112,"threads_started":1}
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:42:58.067512 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:58.061481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:46955)
0807 08:42:58.061590 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:58.061602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:58.061687 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:58.061854 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:58.061855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:58.062018 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:58.062095 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.062098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.063105 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.063106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.063996 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.063999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.064497 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.064499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.064566 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.065078 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:58.065084 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:58.067428 (+ 2344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260807 08:42:58.067636 6269 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:58.061545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:46955 (local address 127.28.108.190:42149)
0807 08:42:58.061628 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:58.061631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:58.061672 (+ 41us) server_negotiation.cc:407] Connection header received
0807 08:42:58.061756 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:58.061757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:58.061771 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:58.061840 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:58.062184 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.062998 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.064110 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.064398 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.064494 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.067566 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:46955: BlockingRecv error: recv got EOF from 127.28.108.129:46955 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260807 08:42:58.067952 6264 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 08:42:58.147357 6268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:58.147647 6268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:58.147691 6268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:58.148957 6268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:58.149039 6268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:58.149089 6268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:42:58.150820 6268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.130.keytab
--principal=oryx/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:38451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=40813
--tserver_master_addrs=127.28.108.190:42149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:58.151155 6268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:58.151418 6268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:58.157379 6274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:58.157378 6275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:58.157596 6268 server_base.cc:1034] running on GCE node
W20260807 08:42:58.157397 6278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:58.157987 6268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:58.158434 6268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:58.159621 6268 hybrid_clock.cc:648] HybridClock initialized: now 1786092178159623 us; error 65 us; skew 500 ppm
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092178, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:58.162680 6268 init.cc:377] Logged in from keytab as oryx/127.28.108.130@KRBTEST.COM (short username oryx)
I20260807 08:42:58.163254 6268 webserver.cc:460] Webserver started at http://127.28.108.130:40813/ using document root <none> and password file <none>
I20260807 08:42:58.163525 6268 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:58.163617 6268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:58.165216 6268 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:42:58.166029 6285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:58.166230 6268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:42:58.166436 6268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "585caeb799d94e6e9e8dd332d262e410"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "bb415ee6f23477981188ffd02f0aacc3"
server_key_iv: "50e0e5c26c698323fb38e1ad6560a7ea"
server_key_version: "encryptionkey@0"
I20260807 08:42:58.166814 6268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:58.175089 6268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:58.177663 6268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:58.177860 6268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:58.178122 6268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:58.178478 6268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:58.178532 6268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:58.178575 6268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:58.178611 6268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:58.185701 6268 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:38451
I20260807 08:42:58.185745 6403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:38451 every 8 connection(s)
I20260807 08:42:58.186168 6268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:42:58.186640 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 6268
I20260807 08:42:58.187074 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:46333
--local_ip_for_outbound_sockets=127.28.108.131
--tserver_master_addrs=127.28.108.190:42149
--webserver_port=34891
--webserver_interface=127.28.108.131
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.108.131
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 08:42:58.194572 6269 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:58.187318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:40449 (local address 127.28.108.190:42149)
0807 08:42:58.187391 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:58.187394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:58.187551 (+ 157us) server_negotiation.cc:407] Connection header received
0807 08:42:58.188163 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:58.188164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:58.188181 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:58.188241 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:58.189318 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.189938 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.190997 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.191207 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.191297 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.194514 (+ 3217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:40449: BlockingRecv error: recv got EOF from 127.28.108.130:40449 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 08:42:58.194633 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:58.187162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:40449)
0807 08:42:58.187384 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:58.187445 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:58.188036 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:58.188322 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:58.188327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:58.188971 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:58.189205 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.189216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.190058 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.190059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.190903 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.190905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.191314 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.191315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.191442 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.192046 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:58.192056 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:58.194405 (+ 2349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
W20260807 08:42:58.195209 6404 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:42149 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.108.190:42149: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:42:58.201562 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:58.195495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:33897)
0807 08:42:58.195550 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:58.195561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:58.195605 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:58.195781 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:58.195782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:58.195980 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:58.196058 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.196060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.196893 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.196894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.197860 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.197864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.198355 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.198357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.198440 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.198994 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:58.199000 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:58.201478 (+ 2478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260807 08:42:58.201793 6269 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:58.195559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:33897 (local address 127.28.108.190:42149)
0807 08:42:58.195601 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:58.195603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:58.195617 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:42:58.195670 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:58.195671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:58.195684 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:58.195753 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:58.196151 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.196797 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.197966 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.198262 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.198334 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.201696 (+ 3362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:33897: BlockingRecv error: recv got EOF from 127.28.108.130:33897 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:42:58.207933 6408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:58.201870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:38423)
0807 08:42:58.202059 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:58.202073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:58.202136 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:58.202299 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:58.202300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:58.202513 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:58.202600 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.202602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.203774 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.203775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.204506 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.204510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.205060 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.205061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.205132 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.205800 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:58.205806 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:58.207823 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
W20260807 08:42:58.208092 6404 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 08:42:58.208107 6269 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:58.202090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:38423 (local address 127.28.108.190:42149)
0807 08:42:58.202153 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:58.202155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:58.202165 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:58.202200 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:58.202201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:58.202210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:58.202273 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:58.202769 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.203600 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.204601 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.204931 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.205004 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.208018 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:38423: BlockingRecv error: recv got EOF from 127.28.108.130:38423 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260807 08:42:58.278566 6407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:42:58.278810 6407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:42:58.278838 6407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:42:58.280181 6407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:42:58.280243 6407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:42:58.280273 6407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:42:58.282094 6407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/krb5kdc/oryx_127.28.108.131.keytab
--principal=oryx/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:46333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=34891
--tserver_master_addrs=127.28.108.190:42149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:42:58.282438 6407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:42:58.282727 6407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:42:58.288435 6414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:42:58.288807 6413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:58.288641 6407 server_base.cc:1034] running on GCE node
W20260807 08:42:58.288435 6416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:42:58.289352 6407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:42:58.289710 6407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:42:58.290977 6407 hybrid_clock.cc:648] HybridClock initialized: now 1786092178290948 us; error 47 us; skew 500 ppm
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092178, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:42:58.294987 6407 init.cc:377] Logged in from keytab as oryx/127.28.108.131@KRBTEST.COM (short username oryx)
I20260807 08:42:58.295634 6407 webserver.cc:460] Webserver started at http://127.28.108.131:34891/ using document root <none> and password file <none>
I20260807 08:42:58.295994 6407 fs_manager.cc:359] Metadata directory not provided
I20260807 08:42:58.296103 6407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:42:58.297959 6407 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260807 08:42:58.298791 6423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:58.299010 6407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:42:58.299113 6407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "30b2b91aad984dbabb2d41fcaa4a11b0"
format_stamp: "Formatted at 2026-08-07 08:42:57 on dist-test-slave-9zdj"
server_key: "ac28df650407e3aead8d6c082c6f1f8e"
server_key_iv: "1b06e8e66c678c8e2965ad5d146cd0be"
server_key_version: "encryptionkey@0"
I20260807 08:42:58.299518 6407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:42:58.328742 6407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:42:58.331461 6407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:42:58.331655 6407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:42:58.331947 6407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:42:58.332293 6407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:42:58.332346 6407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:58.332391 6407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:42:58.332417 6407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:42:58.338485 6407 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:46333
I20260807 08:42:58.338532 6543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:46333 every 8 connection(s)
I20260807 08:42:58.338927 6407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:42:58.345388 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 6407
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:42:58.347196 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:58.339678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:49967)
0807 08:42:58.339907 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:58.339949 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:58.340559 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:58.340843 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:58.340847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:58.341532 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:58.341840 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.341846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.342548 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.342548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.343457 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.343460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.343804 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.343805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.343907 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.344396 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:58.344405 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:58.347037 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
W20260807 08:42:58.347200 6269 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:58.339679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:49967 (local address 127.28.108.190:42149)
0807 08:42:58.339752 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:58.339755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:58.340072 (+ 317us) server_negotiation.cc:407] Connection header received
0807 08:42:58.340663 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:58.340664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:58.340680 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:58.340748 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:58.341947 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.342460 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.343543 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.343731 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.343788 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.347134 (+ 3346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:49967: BlockingRecv error: recv got EOF from 127.28.108.131:49967 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260807 08:42:58.347652 6544 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:42149 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.108.190:42149: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:42:58.353065 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:58.347914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:55769)
0807 08:42:58.347956 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:58.347966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:58.348003 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:58.348161 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:58.348162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:58.348301 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:58.348345 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.348347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.349141 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.349142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.350076 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.350080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.350613 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.350614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.350701 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.351137 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:58.351142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:58.352973 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260807 08:42:58.353116 6269 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:58.347954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:55769 (local address 127.28.108.190:42149)
0807 08:42:58.347998 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:58.347999 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:58.348008 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:42:58.348072 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:58.348073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:58.348083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:58.348154 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:58.348416 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.349043 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.350167 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.350535 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.350615 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.353049 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:55769: BlockingRecv error: recv got EOF from 127.28.108.131:55769 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 08:42:58.353926 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:58.345995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.28.108.190:42149)
0807 08:42:58.346055 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:58.346058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:58.346119 (+ 61us) server_negotiation.cc:407] Connection header received
0807 08:42:58.346213 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:58.346214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:58.346228 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:58.346297 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:58.346928 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.347608 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.348485 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.348775 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.348846 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.350304 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:42:58.350310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:42:58.350310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:42:58.350324 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:42:58.352206 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:58.352580 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:58.352584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:58.352585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:58.352739 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:42:58.352960 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:42:58.352961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:42:58.352962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:42:58.353326 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:42:58.353494 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 08:42:58.353657 (+ 163us) server_negotiation.cc:299] Negotiation successful
0807 08:42:58.353792 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:58 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:42:58.357904 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:58.353470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:37657)
0807 08:42:58.353516 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:58.353526 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:58.353559 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:58.353715 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:58.353716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:58.353869 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:58.353899 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.353901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.354734 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.354736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.355326 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:58.355328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.355727 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.355727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:58.355764 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.356108 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:58.356111 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:58.357856 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260807 08:42:58.358035 6269 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:58.353534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:37657 (local address 127.28.108.190:42149)
0807 08:42:58.353575 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:58.353577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:58.353583 (+ 6us) server_negotiation.cc:407] Connection header received
0807 08:42:58.353604 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:58.353604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:58.353636 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:58.353706 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:58.353956 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.354659 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.355413 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:58.355597 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:58.355728 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:58.357958 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:37657: BlockingRecv error: recv got EOF from 127.28.108.131:37657 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260807 08:42:58.358044 6544 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 08:42:59 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:59 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:42:59.074947 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:59.068547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:58331)
0807 08:42:59.068719 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:59.068731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:59.068836 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:59.069005 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:59.069006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:59.069240 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:59.069298 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:59.069300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.070328 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.070330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:59.071031 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:59.071035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.071592 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.071594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:59.071703 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:59.072546 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:59.072552 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:59.074871 (+ 2319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
W20260807 08:42:59.075089 6549 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:59.068695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:58331 (local address 127.28.108.190:42149)
0807 08:42:59.068857 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:59.068859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:59.068872 (+ 13us) server_negotiation.cc:407] Connection header received
0807 08:42:59.068902 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:59.068902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:59.068914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:59.068984 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:59.069386 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.070232 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.071156 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.071485 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.071559 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:59.074994 (+ 3435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:58331: BlockingRecv error: recv got EOF from 127.28.108.129:58331 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":68,"threads_started":1}
Aug 07 08:42:59 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:59 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:42:59.215193 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:59.208834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:60381)
0807 08:42:59.209027 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:59.209042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:59.209169 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:59.209497 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:59.209499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:59.209731 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:59.209794 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:59.209796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.210623 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.210624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:59.211229 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:59.211231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.211671 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.211672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:59.211751 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:59.212376 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:59.212381 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:59.215093 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":103,"threads_started":1}
W20260807 08:42:59.215272 6549 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:59.208934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:60381 (local address 127.28.108.190:42149)
0807 08:42:59.209017 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:59.209019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:59.209145 (+ 126us) server_negotiation.cc:407] Connection header received
0807 08:42:59.209348 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:59.209350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:59.209366 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:59.209452 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:59.209932 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.210511 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.211340 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.211570 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.211659 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:59.215218 (+ 3559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:60381: BlockingRecv error: recv got EOF from 127.28.108.130:60381 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 07 08:42:59 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:42:59 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:42:59.364557 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:42:59.358598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:60041)
0807 08:42:59.358761 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 08:42:59.358776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:42:59.358846 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:42:59.359086 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:42:59.359087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:42:59.359379 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:42:59.359438 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:59.359440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.360355 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.360356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:59.360982 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:42:59.360984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.361442 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.361443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:42:59.361493 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:59.362019 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:42:59.362025 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:42:59.364480 (+ 2455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":152,"threads_started":1}
W20260807 08:42:59.364782 6549 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:42:59.358751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:60041 (local address 127.28.108.190:42149)
0807 08:42:59.358833 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0807 08:42:59.358835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:42:59.358845 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:42:59.358975 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:42:59.358976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:42:59.358991 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:42:59.359056 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:42:59.359537 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.360149 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.361089 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:42:59.361355 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:42:59.361426 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:42:59.364635 (+ 3209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:60041: BlockingRecv error: recv got EOF from 127.28.108.131:60041 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 07 08:43:00 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:00 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:00.082036 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:00.075634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:33163)
0807 08:43:00.075791 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:00.075805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:00.075879 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:00.076142 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:00.076143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:00.076455 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:00.076512 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:00.076515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.077394 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.077396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:00.078371 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:00.078375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.078844 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.078845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:00.078898 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:00.079491 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:00.079496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:00.081926 (+ 2430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260807 08:43:00.082118 6553 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:00.075805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:33163 (local address 127.28.108.190:42149)
0807 08:43:00.075944 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:00.075946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:00.075956 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:43:00.075985 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:00.075985 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:00.075995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:00.076075 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:00.076633 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.077296 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.078482 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.078704 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.078774 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:00.082041 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:33163: BlockingRecv error: recv got EOF from 127.28.108.129:33163 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
Aug 07 08:43:00 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:00 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:00.222355 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:00.215965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:40983)
0807 08:43:00.216135 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:00.216146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:00.216212 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:00.216527 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:00.216528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:00.216749 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:00.216813 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:00.216815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.217826 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.217828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:00.218568 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:00.218571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.219057 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.219058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:00.219133 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:00.219810 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:00.219815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:00.222286 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
W20260807 08:43:00.222682 6553 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:00.216078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:40983 (local address 127.28.108.190:42149)
0807 08:43:00.216147 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:00.216149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:00.216196 (+ 47us) server_negotiation.cc:407] Connection header received
0807 08:43:00.216406 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:00.216407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:00.216423 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:00.216505 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:00.216897 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.217682 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.218684 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.218903 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.218975 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:00.222630 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:40983: BlockingRecv error: recv got EOF from 127.28.108.130:40983 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 07 08:43:00 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:00 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:00.371802 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:00.365341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:42189)
0807 08:43:00.365545 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:00.365565 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:00.365639 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:00.365895 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:00.365896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:00.366077 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:00.366133 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:00.366134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.366966 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.366967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:00.367623 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:00.367626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.368145 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.368146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:00.368211 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:00.368769 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:00.368775 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:00.371680 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
W20260807 08:43:00.371850 6553 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:00.365514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:42189 (local address 127.28.108.190:42149)
0807 08:43:00.365592 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:00.365594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:00.365761 (+ 167us) server_negotiation.cc:407] Connection header received
0807 08:43:00.365789 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:00.365790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:00.365805 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:00.365852 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:00.366269 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.366851 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.367835 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:00.368041 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:00.368110 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:00.371783 (+ 3673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:42189: BlockingRecv error: recv got EOF from 127.28.108.131:42189 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 07 08:43:01 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:01 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:01.089082 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:01.082794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:42515)
0807 08:43:01.082984 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:01.082995 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:01.083065 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:01.083278 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:01.083279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:01.083538 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:01.083643 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:01.083647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.084654 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.084655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:01.085334 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:01.085337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.085963 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.085964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:01.086024 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:01.086709 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:01.086716 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:01.088996 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
W20260807 08:43:01.089383 6558 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:01.082897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:42515 (local address 127.28.108.190:42149)
0807 08:43:01.083066 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:01.083068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:01.083128 (+ 60us) server_negotiation.cc:407] Connection header received
0807 08:43:01.083157 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:01.083158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:01.083170 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:01.083258 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:01.083762 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.084536 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.085531 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.085853 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.085928 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:01.089318 (+ 3390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:42515: BlockingRecv error: recv got EOF from 127.28.108.129:42515 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
Aug 07 08:43:01 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:01 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:01.228731 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:01.222965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:37973)
0807 08:43:01.223147 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:01.223159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:01.223220 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:01.223354 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:01.223355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:01.223537 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:01.223592 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:01.223593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.224355 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.224356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:01.224974 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:01.224977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.225523 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.225524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:01.225575 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:01.226245 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:01.226251 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:01.228664 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
W20260807 08:43:01.229246 6558 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:01.223031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:37973 (local address 127.28.108.190:42149)
0807 08:43:01.223105 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:01.223107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:01.223223 (+ 116us) server_negotiation.cc:407] Connection header received
0807 08:43:01.223281 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:01.223282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:01.223294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:01.223345 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:01.223666 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.224241 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.225083 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.225426 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.225505 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:01.229096 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:37973: BlockingRecv error: recv got EOF from 127.28.108.130:37973 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 07 08:43:01 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:01 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:01.378091 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:01.372451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:49009)
0807 08:43:01.372614 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:01.372632 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:01.372698 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:01.372966 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:01.372968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:01.373176 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:01.373235 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:01.373237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.374104 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.374105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:01.374721 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:01.374725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.375132 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.375133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:01.375200 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:01.375890 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:01.375895 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:01.378010 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
W20260807 08:43:01.378333 6558 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:01.372536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:49009 (local address 127.28.108.190:42149)
0807 08:43:01.372628 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:01.372631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:01.372760 (+ 129us) server_negotiation.cc:407] Connection header received
0807 08:43:01.372840 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:01.372841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:01.372855 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:01.372921 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:01.373463 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.374025 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.374816 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:01.375037 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:01.375098 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:01.378256 (+ 3158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:49009: BlockingRecv error: recv got EOF from 127.28.108.131:49009 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 08:43:02 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:02 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:02.100625 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:02.089911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:33385)
0807 08:43:02.090090 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:02.090108 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:02.090171 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:02.090387 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:02.090388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:02.090583 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:02.090644 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:02.090646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.091721 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.091723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:02.092469 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:02.092475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.092982 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.092983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:02.093046 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:02.098175 (+ 5129us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:02.098185 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:02.100549 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":118,"threads_started":1}
W20260807 08:43:02.100865 6562 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:02.089997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:33385 (local address 127.28.108.190:42149)
0807 08:43:02.090184 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:02.090187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:02.090198 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:43:02.090229 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:02.090230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:02.090246 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:02.090334 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:02.090833 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.091508 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.092602 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.092869 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.092943 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:02.100776 (+ 7833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:33385: BlockingRecv error: recv got EOF from 127.28.108.129:33385 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":248,"threads_started":1}
Aug 07 08:43:02 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:02 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:02.235596 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:02.229522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:51351)
0807 08:43:02.229764 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:02.229776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:02.229851 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:02.230041 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:02.230042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:02.230344 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:02.230403 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:02.230405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.231369 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.231370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:02.231922 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:02.231923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.232367 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.232368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:02.232426 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:02.233078 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:02.233083 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:02.235522 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":125,"threads_started":1}
W20260807 08:43:02.235769 6562 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:02.229672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:51351 (local address 127.28.108.190:42149)
0807 08:43:02.229762 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:02.229764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:02.229868 (+ 104us) server_negotiation.cc:407] Connection header received
0807 08:43:02.229905 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:02.229906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:02.229921 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:02.229986 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:02.230514 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.231151 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.232042 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.232252 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.232332 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:02.235685 (+ 3353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:51351: BlockingRecv error: recv got EOF from 127.28.108.130:51351 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 07 08:43:02 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:02 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:02.384774 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:02.378699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:36211)
0807 08:43:02.378933 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:02.378954 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:02.379042 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:02.379273 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:02.379274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:02.379506 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:02.379569 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:02.379571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.380609 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.380610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:02.381166 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:02.381168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.381812 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.381813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:02.381878 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:02.382529 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:02.382535 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:02.384707 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":148,"threads_started":1}
W20260807 08:43:02.385111 6562 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:02.378860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:36211 (local address 127.28.108.190:42149)
0807 08:43:02.378935 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:02.378937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:02.379070 (+ 133us) server_negotiation.cc:407] Connection header received
0807 08:43:02.379114 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:02.379115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:02.379133 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:02.379213 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:02.379706 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.380413 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.381287 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:02.381599 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:02.381750 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:02.385007 (+ 3257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:36211: BlockingRecv error: recv got EOF from 127.28.108.131:36211 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 07 08:43:03 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:03 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:03.108533 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:03.101295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:58221)
0807 08:43:03.101606 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:03.101672 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:03.101758 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:03.102018 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:03.102019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:03.102261 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:03.102331 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:03.102334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.103535 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.103536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:03.104258 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:03.104261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.104942 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.104943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:03.105010 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:03.105792 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:03.105798 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:03.108429 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":198,"threads_started":1}
W20260807 08:43:03.108608 6566 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:03.101481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:58221 (local address 127.28.108.190:42149)
0807 08:43:03.101761 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:03.101763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:03.101806 (+ 43us) server_negotiation.cc:407] Connection header received
0807 08:43:03.101836 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:03.101837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:03.101850 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:03.101971 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:03.102587 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.103392 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.104543 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.104806 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.104888 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:03.108533 (+ 3645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:58221: BlockingRecv error: recv got EOF from 127.28.108.129:58221 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
Aug 07 08:43:03 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:03 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:03.241950 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:03.236189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:47879)
0807 08:43:03.236410 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:03.236425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:03.236495 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:03.236765 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:03.236766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:03.237008 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:03.237071 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:03.237073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.237899 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.237900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:03.238450 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:03.238452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.238844 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.238845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:03.238907 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:03.239489 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:03.239494 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:03.241876 (+ 2382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
W20260807 08:43:03.242255 6566 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:03.236508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:47879 (local address 127.28.108.190:42149)
0807 08:43:03.236589 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:03.236592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:03.236605 (+ 13us) server_negotiation.cc:407] Connection header received
0807 08:43:03.236630 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:03.236631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:03.236641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:03.236757 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:03.237172 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.237795 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.238548 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.238762 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.238847 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:03.242089 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:47879: BlockingRecv error: recv got EOF from 127.28.108.130:47879 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 07 08:43:03 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:03 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:03.391347 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:03.385473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:49123)
0807 08:43:03.385700 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:03.385716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:03.385813 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:03.386011 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:03.386012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:03.386230 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:03.386284 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:03.386286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.387083 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.387085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:03.387742 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:03.387744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.388171 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.388172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:03.388251 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:03.388825 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:03.388831 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:03.391232 (+ 2401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":119,"threads_started":1}
W20260807 08:43:03.391423 6566 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:03.385650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:49123 (local address 127.28.108.190:42149)
0807 08:43:03.385758 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:03.385761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:03.385773 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:43:03.385924 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:03.385925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:03.385942 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:03.386005 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:03.386400 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.386977 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.387846 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:03.388075 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:03.388167 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:03.391350 (+ 3183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:49123: BlockingRecv error: recv got EOF from 127.28.108.131:49123 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 07 08:43:04 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:04 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:04.116405 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:04.109413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:50337)
0807 08:43:04.109645 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:04.109660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:04.109750 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:04.109987 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:04.109987 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:04.110266 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:04.110350 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:04.110356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.111174 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.111175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:04.111923 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:04.111926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.112483 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.112484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:04.112561 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:04.113299 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:04.113304 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:04.116292 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
W20260807 08:43:04.116489 6570 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:04.109571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:50337 (local address 127.28.108.190:42149)
0807 08:43:04.109788 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:04.109791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:04.109808 (+ 17us) server_negotiation.cc:407] Connection header received
0807 08:43:04.109846 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:04.109847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:04.109860 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:04.109941 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:04.110464 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.111053 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.112060 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.112388 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.112461 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:04.116405 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:50337: BlockingRecv error: recv got EOF from 127.28.108.129:50337 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":114,"threads_started":1}
Aug 07 08:43:04 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:04 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:04.248699 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:04.242571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:48789)
0807 08:43:04.242732 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:04.242755 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:04.242829 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:04.243047 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:04.243048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:04.243254 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:04.243327 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:04.243331 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.244121 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.244122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:04.245027 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:04.245030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.245557 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.245558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:04.245641 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:04.246328 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:04.246334 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:04.248633 (+ 2299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
W20260807 08:43:04.248980 6570 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:04.242647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:48789 (local address 127.28.108.190:42149)
0807 08:43:04.242723 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:04.242725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:04.242819 (+ 94us) server_negotiation.cc:407] Connection header received
0807 08:43:04.242892 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:04.242893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:04.242906 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:04.242970 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:04.243441 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.244021 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.245171 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.245429 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.245511 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:04.248922 (+ 3411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:48789: BlockingRecv error: recv got EOF from 127.28.108.130:48789 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 07 08:43:04 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:04 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:04.398976 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:04.392102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:39943)
0807 08:43:04.392443 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:04.392457 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:04.392531 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:04.392818 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:04.392819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:04.393038 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:04.393105 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:04.393107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.394147 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.394149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:04.394967 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:04.394971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.395534 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.395535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:04.395601 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:04.396219 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:04.396224 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:04.398904 (+ 2680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":242,"threads_started":1}
W20260807 08:43:04.399250 6570 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:04.392276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:39943 (local address 127.28.108.190:42149)
0807 08:43:04.392633 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:04.392635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:04.392646 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:43:04.392676 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:04.392677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:04.392693 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:04.392765 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:04.393213 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.394022 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.395093 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:04.395411 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:04.395492 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:04.399088 (+ 3596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:39943: BlockingRecv error: recv got EOF from 127.28.108.131:39943 (error 108)
Metrics: {"server-negotiator.queue_time_us":317}
Aug 07 08:43:05 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:05 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:05.123251 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:05.117020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:59989)
0807 08:43:05.117165 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:05.117182 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:05.117277 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:05.117600 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:05.117601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:05.117861 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:05.117922 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:05.117925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.118787 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.118787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:05.119342 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:05.119344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.119768 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.119769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:05.119826 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:05.120545 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:05.120551 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:05.123152 (+ 2601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
W20260807 08:43:05.123338 6574 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:05.117139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:59989 (local address 127.28.108.190:42149)
0807 08:43:05.117437 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:05.117439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:05.117450 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:43:05.117479 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:05.117480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:05.117492 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:05.117559 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:05.118004 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.118618 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.119433 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.119648 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.119706 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:05.123254 (+ 3548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:59989: BlockingRecv error: recv got EOF from 127.28.108.129:59989 (error 108)
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":211,"threads_started":1}
Aug 07 08:43:05 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:05 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:05.255662 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:05.249317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:48819)
0807 08:43:05.249497 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:05.249512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:05.249583 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:05.249805 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:05.249806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:05.249989 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:05.250046 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:05.250048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.251295 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.251297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:05.251893 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:05.251896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.252505 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.252506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:05.252567 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:05.253118 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:05.253124 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:05.255571 (+ 2447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":110,"threads_started":1}
W20260807 08:43:05.255797 6574 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:05.249471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:48819 (local address 127.28.108.190:42149)
0807 08:43:05.249565 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:05.249567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:05.249578 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:43:05.249707 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:05.249708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:05.249720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:05.249783 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:05.250148 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.251070 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.252036 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.252385 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.252469 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:05.255730 (+ 3261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:48819: BlockingRecv error: recv got EOF from 127.28.108.130:48819 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 08:43:05 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:05 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:05.405530 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:05.399625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:35247)
0807 08:43:05.399863 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:05.399876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:05.399947 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:05.400107 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:05.400108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:05.400291 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:05.400346 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:05.400348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.401115 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.401117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:05.401984 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:05.401987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.402498 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.402499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:05.402553 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:05.403194 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:05.403199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:05.405464 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":104,"threads_started":1}
W20260807 08:43:05.405772 6574 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:05.399836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:35247 (local address 127.28.108.190:42149)
0807 08:43:05.399890 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:05.399892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:05.399935 (+ 43us) server_negotiation.cc:407] Connection header received
0807 08:43:05.400010 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:05.400011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:05.400024 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:05.400082 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:05.400436 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.400990 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.402086 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:05.402414 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:05.402487 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:05.405702 (+ 3215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:35247: BlockingRecv error: recv got EOF from 127.28.108.131:35247 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 07 08:43:06 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:06 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:06.130311 6577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:06.124143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:50669)
0807 08:43:06.124360 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:06.124374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:06.124472 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:06.124709 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:06.124710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:06.124979 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:06.125039 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:06.125041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.125880 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.125881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:06.126601 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:06.126604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.127095 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.127096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:06.127155 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:06.127778 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:06.127784 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:06.130242 (+ 2458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":120,"threads_started":1}
W20260807 08:43:06.130612 6578 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:06.124299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:50669 (local address 127.28.108.190:42149)
0807 08:43:06.124469 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:06.124472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:06.124483 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:43:06.124572 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:06.124572 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:06.124586 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:06.124656 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:06.125180 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.125774 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.126732 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.126984 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.127064 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:06.130538 (+ 3474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:50669: BlockingRecv error: recv got EOF from 127.28.108.129:50669 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
Aug 07 08:43:06 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:06 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:06.262768 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:06.256568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:34097)
0807 08:43:06.256740 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:06.256765 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:06.256822 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:06.257080 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:06.257081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:06.257312 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:06.257380 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:06.257383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.258393 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.258394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:06.258955 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:06.258957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.259365 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.259366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:06.259429 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:06.260059 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:06.260064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:06.262640 (+ 2576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":107,"threads_started":1}
W20260807 08:43:06.262821 6578 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:06.256829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:34097 (local address 127.28.108.190:42149)
0807 08:43:06.256926 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:06.256936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:06.256948 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:43:06.256986 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:06.256987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:06.256998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:06.257070 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:06.257509 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.258280 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.259051 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.259282 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.259366 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:06.262746 (+ 3380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:34097: BlockingRecv error: recv got EOF from 127.28.108.130:34097 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 08:43:06 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:06 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:06.412417 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:06.406187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:47439)
0807 08:43:06.406431 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:06.406452 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:06.406525 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:06.406720 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:06.406721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:06.406932 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:06.406987 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:06.406989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.407821 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.407822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:06.408445 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:06.408449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.408840 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.408841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:06.408888 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:06.409727 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:06.409732 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:06.412351 (+ 2619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":107,"threads_started":1}
W20260807 08:43:06.412581 6578 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:06.406460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:47439 (local address 127.28.108.190:42149)
0807 08:43:06.406568 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:06.406570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:06.406581 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:43:06.406619 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:06.406620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:06.406634 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:06.406689 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:06.407080 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.407680 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.408551 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:06.408750 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:06.408828 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:06.412486 (+ 3658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:47439: BlockingRecv error: recv got EOF from 127.28.108.131:47439 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 07 08:43:07 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:07 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:07.144516 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:07.131155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:58687)
0807 08:43:07.131405 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:07.131419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:07.131484 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:07.131763 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:07.131764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:07.131962 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:07.132029 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:07.132032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.132988 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.132989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:07.133884 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:07.133889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.134399 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.134400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:07.134465 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:07.141872 (+ 7407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:07.141879 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:07.144443 (+ 2564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":126,"threads_started":1}
W20260807 08:43:07.144868 6583 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:07.131287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:58687 (local address 127.28.108.190:42149)
0807 08:43:07.131499 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:07.131501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:07.131512 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:43:07.131540 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:07.131541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:07.131553 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:07.131620 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:07.132158 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.132867 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.134025 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.134274 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.134364 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:07.144769 (+ 10405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:58687: BlockingRecv error: recv got EOF from 127.28.108.129:58687 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":61,"threads_started":1}
Aug 07 08:43:07 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:07 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:07.270726 6585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:07.263503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:55141)
0807 08:43:07.263731 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:07.263753 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:07.263859 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:07.264055 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:07.264056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:07.264360 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:07.264425 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:07.264427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.265431 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.265432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:07.266640 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:07.266645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.267120 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.267121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:07.267191 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:07.267733 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:07.267738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:07.270632 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":168,"threads_started":1}
W20260807 08:43:07.270845 6583 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:07.263612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:55141 (local address 127.28.108.190:42149)
0807 08:43:07.263787 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:07.263796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:07.263838 (+ 42us) server_negotiation.cc:407] Connection header received
0807 08:43:07.263935 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:07.263936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:07.263949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:07.263999 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:07.264546 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.265314 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.266739 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.267004 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.267075 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:07.270762 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:55141: BlockingRecv error: recv got EOF from 127.28.108.130:55141 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Aug 07 08:43:07 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:07 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:07.419612 6586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:07.413156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:59343)
0807 08:43:07.413343 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:07.413360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:07.413454 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:07.413653 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:07.413654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:07.413914 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:07.413972 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:07.413974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.414859 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.414863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:07.415876 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:07.415880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.416391 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.416392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:07.416449 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:07.416961 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:07.416967 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:07.419544 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":103,"threads_started":1}
W20260807 08:43:07.419883 6583 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:07.413373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:59343 (local address 127.28.108.190:42149)
0807 08:43:07.413476 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:07.413478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:07.413490 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:43:07.413521 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:07.413522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:07.413538 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:07.413742 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:07.414094 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.414712 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.416006 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:07.416293 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:07.416355 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:07.419813 (+ 3458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:59343: BlockingRecv error: recv got EOF from 127.28.108.131:59343 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 07 08:43:08 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:08 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:08.151391 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:08.145441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:53237)
0807 08:43:08.145720 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:08.145733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:08.145822 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:08.146037 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:08.146037 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:08.146221 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:08.146369 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:08.146372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.147363 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.147365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:08.147970 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:08.147973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.148394 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.148395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:08.148443 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:08.148990 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:08.148996 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:08.151220 (+ 2224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":240,"threads_started":1}
W20260807 08:43:08.151418 6588 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:08.145557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:53237 (local address 127.28.108.190:42149)
0807 08:43:08.145741 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:08.145743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:08.145880 (+ 137us) server_negotiation.cc:407] Connection header received
0807 08:43:08.145919 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:08.145921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:08.145935 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:08.146015 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:08.146498 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.147128 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.148062 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.148286 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.148361 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:08.151360 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:53237: BlockingRecv error: recv got EOF from 127.28.108.129:53237 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":121,"threads_started":1}
Aug 07 08:43:08 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:08 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:08.277886 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:08.271480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:50149)
0807 08:43:08.271681 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:08.271708 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:08.271773 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:08.272001 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:08.272002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:08.272197 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:08.272252 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:08.272254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.273331 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.273333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:08.274109 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:08.274113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.274596 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.274597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:08.274687 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:08.275219 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:08.275225 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:08.277781 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
W20260807 08:43:08.277977 6588 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:08.271618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:50149 (local address 127.28.108.190:42149)
0807 08:43:08.271717 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:08.271729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:08.271758 (+ 29us) server_negotiation.cc:407] Connection header received
0807 08:43:08.271845 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:08.271846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:08.271858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:08.271929 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:08.272375 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.272976 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.274233 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.274499 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.274597 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:08.277911 (+ 3314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:50149: BlockingRecv error: recv got EOF from 127.28.108.130:50149 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 08:43:08 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:08 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:08.426802 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:08.420191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:51765)
0807 08:43:08.420390 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:08.420404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:08.420466 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:08.420646 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:08.420647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:08.420976 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:08.421051 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:08.421055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.422124 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.422125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:08.422842 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:08.422845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.423451 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.423452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:08.423516 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:08.424073 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:08.424079 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:08.426672 (+ 2593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":109,"threads_started":1}
W20260807 08:43:08.426920 6588 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:08.420392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:51765 (local address 127.28.108.190:42149)
0807 08:43:08.420461 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:08.420463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:08.420474 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:43:08.420536 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:08.420537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:08.420553 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:08.420624 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:08.421162 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.421953 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.423000 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:08.423312 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:08.423394 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:08.426839 (+ 3445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:51765: BlockingRecv error: recv got EOF from 127.28.108.131:51765 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 07 08:43:09 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:09 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:09.157840 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:09.152100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:34121)
0807 08:43:09.152277 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:09.152289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:09.152357 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:09.152521 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:09.152522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:09.152706 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:09.152805 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:09.152808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.153801 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.153802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:09.154502 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:09.154506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.154964 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.154965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:09.155026 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:09.155571 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:09.155576 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:09.157761 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":111,"threads_started":1}
W20260807 08:43:09.158030 6592 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:09.152204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:34121 (local address 127.28.108.190:42149)
0807 08:43:09.152356 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:09.152358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:09.152398 (+ 40us) server_negotiation.cc:407] Connection header received
0807 08:43:09.152424 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:09.152424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:09.152435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:09.152492 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:09.152924 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.153597 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.154622 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.154848 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.154920 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:09.157941 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:34121: BlockingRecv error: recv got EOF from 127.28.108.129:34121 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
Aug 07 08:43:09 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:09 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:09.284492 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:09.278571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:50081)
0807 08:43:09.278773 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:09.278796 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:09.278888 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:09.279077 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:09.279079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:09.279290 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:09.279348 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:09.279350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.280100 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.280101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:09.280806 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:09.280810 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.281335 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.281336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:09.281399 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:09.282069 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:09.282075 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:09.284421 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":136,"threads_started":1}
W20260807 08:43:09.284724 6592 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:09.278713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:50081 (local address 127.28.108.190:42149)
0807 08:43:09.278780 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:09.278782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:09.278872 (+ 90us) server_negotiation.cc:407] Connection header received
0807 08:43:09.278961 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:09.278962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:09.278975 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:09.279023 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:09.279450 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.279996 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.280928 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.281216 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.281299 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:09.284643 (+ 3344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:50081: BlockingRecv error: recv got EOF from 127.28.108.130:50081 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 07 08:43:09 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:09 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:09.433993 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:09.427578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:59075)
0807 08:43:09.427846 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:09.427861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:09.427990 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:09.428194 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:09.428195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:09.428534 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:09.428630 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:09.428633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.429831 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.429832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:09.430420 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:09.430423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.430763 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.430764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:09.430816 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:09.431440 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:09.431449 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:09.433910 (+ 2461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":141,"threads_started":1}
W20260807 08:43:09.434173 6592 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:09.427720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:59075 (local address 127.28.108.190:42149)
0807 08:43:09.427823 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:09.427826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:09.427967 (+ 141us) server_negotiation.cc:407] Connection header received
0807 08:43:09.428075 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:09.428077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:09.428095 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:09.428187 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:09.428806 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.429740 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.430494 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:09.430686 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:09.430746 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:09.434042 (+ 3296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:59075: BlockingRecv error: recv got EOF from 127.28.108.131:59075 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 08:43:10 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:10 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:10.164436 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:10.158642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:33859)
0807 08:43:10.158820 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:10.158836 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:10.158912 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:10.159157 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:10.159158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:10.159458 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:10.159518 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:10.159520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.160423 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.160424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:10.161028 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:10.161029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.161488 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.161489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:10.161545 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:10.162077 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:10.162082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:10.164361 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":168,"threads_started":1}
W20260807 08:43:10.164686 6597 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:10.158764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:33859 (local address 127.28.108.190:42149)
0807 08:43:10.158968 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:10.158971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:10.158981 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:43:10.159011 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:10.159011 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:10.159022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:10.159125 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:10.159658 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.160323 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.161144 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.161387 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.161457 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:10.164592 (+ 3135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:33859: BlockingRecv error: recv got EOF from 127.28.108.129:33859 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
Aug 07 08:43:10 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:10 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:10.291296 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:10.285421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:40521)
0807 08:43:10.285695 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:10.285708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:10.285799 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:10.285987 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:10.285988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:10.286207 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:10.286259 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:10.286261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.287081 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.287083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:10.287639 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:10.287641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.288051 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.288053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:10.288116 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:10.288677 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:10.288682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:10.291171 (+ 2489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":131,"threads_started":1}
W20260807 08:43:10.291376 6597 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:10.285557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:40521 (local address 127.28.108.190:42149)
0807 08:43:10.285675 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:10.285687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:10.285766 (+ 79us) server_negotiation.cc:407] Connection header received
0807 08:43:10.285874 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:10.285876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:10.285893 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:10.285977 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:10.286368 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.286944 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.287744 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.287961 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.288051 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:10.291305 (+ 3254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:40521: BlockingRecv error: recv got EOF from 127.28.108.130:40521 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 07 08:43:10 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:10 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:10.440903 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:10.434650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:41159)
0807 08:43:10.434925 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:10.434937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:10.435046 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:10.435242 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:10.435243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:10.435488 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:10.435554 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:10.435559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.436314 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.436315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:10.436892 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:10.436894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.437300 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.437301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:10.437367 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:10.438080 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:10.438086 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:10.440746 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":153,"threads_started":1}
W20260807 08:43:10.440939 6597 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:10.434799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:41159 (local address 127.28.108.190:42149)
0807 08:43:10.434904 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:10.434907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:10.435018 (+ 111us) server_negotiation.cc:407] Connection header received
0807 08:43:10.435131 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:10.435132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:10.435151 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:10.435232 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:10.435651 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.436198 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.437005 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:10.437209 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:10.437300 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:10.440867 (+ 3567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:41159: BlockingRecv error: recv got EOF from 127.28.108.131:41159 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 08:43:11 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:11 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:11.171622 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:11.165097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:56645)
0807 08:43:11.165279 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:11.165292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:11.165362 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:11.165685 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:11.165686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:11.165901 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:11.165957 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:11.165960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.166934 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.166935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:11.167606 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:11.167609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.168085 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.168086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:11.168148 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:11.168901 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:11.168906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:11.171545 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":120,"threads_started":1}
W20260807 08:43:11.171888 6601 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:11.165202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:56645 (local address 127.28.108.190:42149)
0807 08:43:11.165451 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:11.165454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:11.165465 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:43:11.165503 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:11.165504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:11.165516 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:11.165594 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:11.166068 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.166775 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.167742 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.167972 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.168045 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:11.171805 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:56645: BlockingRecv error: recv got EOF from 127.28.108.129:56645 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":187,"threads_started":1}
Aug 07 08:43:11 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:11 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:11.297915 6602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:11.291947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:46007)
0807 08:43:11.292132 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:11.292145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:11.292199 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:11.292399 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:11.292400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:11.292625 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:11.292747 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:11.292749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.293568 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.293569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:11.294432 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:11.294435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.294931 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.294932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:11.295014 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:11.295637 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:11.295642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:11.297850 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":115,"threads_started":1}
W20260807 08:43:11.298260 6601 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:11.292156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:46007 (local address 127.28.108.190:42149)
0807 08:43:11.292213 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:11.292216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:11.292225 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:43:11.292256 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:11.292256 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:11.292269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:11.292340 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:11.292873 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.293447 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.294554 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.294783 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.294868 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:11.298094 (+ 3226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:46007: BlockingRecv error: recv got EOF from 127.28.108.130:46007 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 07 08:43:11 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:11 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:11.447599 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:11.441532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:43853)
0807 08:43:11.441722 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:11.441741 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:11.441813 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:11.441974 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:11.441975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:11.442307 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:11.442367 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:11.442369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.443254 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.443255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:11.443844 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:11.443847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.444274 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.444275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:11.444429 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:11.444984 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:11.444990 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:11.447509 (+ 2519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":145,"threads_started":1}
W20260807 08:43:11.447804 6601 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:11.441694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:43853 (local address 127.28.108.190:42149)
0807 08:43:11.441742 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:11.441744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:11.441797 (+ 53us) server_negotiation.cc:407] Connection header received
0807 08:43:11.441867 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:11.441868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:11.441884 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:11.441952 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:11.442509 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.443084 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.443953 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:11.444163 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:11.444229 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:11.447702 (+ 3473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:43853: BlockingRecv error: recv got EOF from 127.28.108.131:43853 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 07 08:43:12 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:12 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:12.188776 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:12.172154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:52711)
0807 08:43:12.172306 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:12.172320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:12.172386 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:12.172523 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:12.172524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:12.172707 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:12.172804 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:12.172807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.173560 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.173562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:12.174242 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:12.174245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.174760 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.174761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:12.174814 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:12.186321 (+ 11507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:12.186329 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:12.188671 (+ 2342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
W20260807 08:43:12.189043 6605 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:12.172239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:52711 (local address 127.28.108.190:42149)
0807 08:43:12.172390 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:12.172393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:12.172402 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:43:12.172445 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:12.172445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:12.172456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:12.172511 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:12.172886 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.173454 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.174416 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.174649 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.174726 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:12.188962 (+ 14236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:52711: BlockingRecv error: recv got EOF from 127.28.108.129:52711 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
Aug 07 08:43:12 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:12 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:12.304970 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:12.298553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:44263)
0807 08:43:12.298751 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:12.298827 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:12.298986 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:12.299179 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:12.299180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:12.299383 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:12.299435 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:12.299437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.300274 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.300275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:12.300948 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:12.300951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.301446 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.301448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:12.301527 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:12.302206 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:12.302213 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:12.304900 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
W20260807 08:43:12.305178 6605 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:12.298687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:44263 (local address 127.28.108.190:42149)
0807 08:43:12.298746 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:12.298749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:12.298981 (+ 232us) server_negotiation.cc:407] Connection header received
0807 08:43:12.299075 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:12.299076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:12.299091 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:12.299148 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:12.299513 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.300064 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.301067 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.301319 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.301406 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:12.305051 (+ 3645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:44263: BlockingRecv error: recv got EOF from 127.28.108.130:44263 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 07 08:43:12 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:12 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.131@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:12.455056 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:12.448457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.131:36831)
0807 08:43:12.448654 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:12.448745 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:12.448820 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:12.449095 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:12.449096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:12.449450 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:12.449506 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:12.449509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.450288 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.450289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:12.450893 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:12.450896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.451432 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.451433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:12.451494 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:12.452192 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:12.452199 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:12.454943 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":130,"threads_started":1}
W20260807 08:43:12.455148 6605 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:12.448577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:36831 (local address 127.28.108.190:42149)
0807 08:43:12.448679 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:12.448682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:12.448847 (+ 165us) server_negotiation.cc:407] Connection header received
0807 08:43:12.448884 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:12.448886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:12.448904 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:12.448972 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:12.449647 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.450194 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.450994 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:12.451317 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:12.451384 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:12.455067 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.131:36831: BlockingRecv error: recv got EOF from 127.28.108.131:36831 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 08:43:13 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:13 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.129@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:13.195554 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:13.189493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.129:59479)
0807 08:43:13.189788 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:13.189799 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:13.189863 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:13.190082 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:13.190083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:13.190372 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:13.190429 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:13.190431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.191316 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.191317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:13.191905 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:13.191907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.192370 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.192371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:13.192434 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:13.193053 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:13.193058 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:13.195474 (+ 2416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":90,"threads_started":1}
W20260807 08:43:13.195870 6609 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:13.189771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:59479 (local address 127.28.108.190:42149)
0807 08:43:13.189932 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:13.189934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:13.189952 (+ 18us) server_negotiation.cc:407] Connection header received
0807 08:43:13.189980 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:13.189981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:13.189993 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:13.190061 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:13.190547 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.191121 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.191999 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.192245 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.192324 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:13.195757 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.129:59479: BlockingRecv error: recv got EOF from 127.28.108.129:59479 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
Aug 07 08:43:13 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 08:43:13 dist-test-slave-9zdj krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.108.130@KRBTEST.COM for oryx/127.28.108.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 08:43:13.311589 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:13.305812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42149 (local address 127.28.108.130:56665)
0807 08:43:13.305961 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:13.305973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:13.306035 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:13.306235 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:13.306237 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:13.306541 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:13.306597 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:13.306600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.307508 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.307509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:13.308142 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:13.308144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.308562 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.308563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:13.308615 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:13.309156 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:13.309161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:13.311516 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.108.190:42149: Server oryx/127.28.108.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
W20260807 08:43:13.311721 6609 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:13.305943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:56665 (local address 127.28.108.190:42149)
0807 08:43:13.306017 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:13.306019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:13.306032 (+ 13us) server_negotiation.cc:407] Connection header received
0807 08:43:13.306096 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:13.306097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:13.306108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:13.306176 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:13.306781 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.307404 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.308251 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.308478 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.308547 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:13.311666 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.108.130:56665: BlockingRecv error: recv got EOF from 127.28.108.130:56665 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 08:43:13.346975 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 6103
I20260807 08:43:13.355372 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 6268
I20260807 08:43:13.363363 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 6407
I20260807 08:43:13.370733 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 6025
2026-08-07T08:43:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17894 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T08:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:43:13Z Disabled control of system clock
I20260807 08:43:13.406953 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:45113
--webserver_interface=127.28.108.190
--webserver_port=0
--builtin_ntp_servers=127.28.108.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:45113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 08:43:13.496697 6615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:13.496973 6615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:13.496999 6615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:13.498409 6615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:43:13.498471 6615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:13.498483 6615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:13.498497 6615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:43:13.498510 6615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:43:13.500097 6615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:45113
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:45113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:13.500373 6615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:13.500664 6615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:13.506944 6620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:13.506944 6623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:13.506989 6621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:13.507221 6615 server_base.cc:1034] running on GCE node
I20260807 08:43:13.507565 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:13.507965 6615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:13.509286 6615 hybrid_clock.cc:648] HybridClock initialized: now 1786092193509160 us; error 44 us; skew 500 ppm
I20260807 08:43:13.510970 6615 webserver.cc:460] Webserver started at http://127.28.108.190:40065/ using document root <none> and password file <none>
I20260807 08:43:13.511395 6615 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:13.511454 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:13.511632 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:13.512912 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "e9f50adcd06148e285ccb354c946995d"
format_stamp: "Formatted at 2026-08-07 08:43:13 on dist-test-slave-9zdj"
server_key: "847185dd1737c656eeb0e96e558aec09"
server_key_iv: "b7db751af136d3a628340e0bd94ca8be"
server_key_version: "encryptionkey@0"
I20260807 08:43:13.513329 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "e9f50adcd06148e285ccb354c946995d"
format_stamp: "Formatted at 2026-08-07 08:43:13 on dist-test-slave-9zdj"
server_key: "847185dd1737c656eeb0e96e558aec09"
server_key_iv: "b7db751af136d3a628340e0bd94ca8be"
server_key_version: "encryptionkey@0"
I20260807 08:43:13.514990 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 08:43:13.516263 6630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:13.516549 6615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:13.516660 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "e9f50adcd06148e285ccb354c946995d"
format_stamp: "Formatted at 2026-08-07 08:43:13 on dist-test-slave-9zdj"
server_key: "847185dd1737c656eeb0e96e558aec09"
server_key_iv: "b7db751af136d3a628340e0bd94ca8be"
server_key_version: "encryptionkey@0"
I20260807 08:43:13.516721 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:13.527139 6615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:13.527602 6615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:13.527815 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:13.532473 6615 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:45113
I20260807 08:43:13.532637 6685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:45113 every 8 connection(s)
I20260807 08:43:13.532994 6615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:43:13.534034 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:43:13.537320 6686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d: Bootstrap starting.
I20260807 08:43:13.538141 6686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d: Neither blocks nor log segments found. Creating new log.
I20260807 08:43:13.538583 6686 log.cc:824] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d: Log is configured to *not* fsync() on all Append() calls
I20260807 08:43:13.539758 6686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d: No bootstrap required, opened a new log
I20260807 08:43:13.541887 6686 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f50adcd06148e285ccb354c946995d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 45113 } }
I20260807 08:43:13.542079 6686 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:43:13.542104 6686 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f50adcd06148e285ccb354c946995d, State: Initialized, Role: FOLLOWER
I20260807 08:43:13.542300 6686 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f50adcd06148e285ccb354c946995d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 45113 } }
I20260807 08:43:13.542387 6686 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:43:13.542414 6686 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:43:13.542445 6686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:43:13.542984 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 6615
I20260807 08:43:13.543064 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:43:13.543267 6686 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f50adcd06148e285ccb354c946995d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 45113 } }
I20260807 08:43:13.543367 29106 external_mini_cluster.cc:1428] Setting key ae5baff73d1dec7cc49ac3447fa0c623
I20260807 08:43:13.543458 6686 leader_election.cc:302] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9f50adcd06148e285ccb354c946995d; no voters:
I20260807 08:43:13.544009 6689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:43:13.544200 6689 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [term 1 LEADER]: Becoming Leader. State: Replica: e9f50adcd06148e285ccb354c946995d, State: Running, Role: LEADER
I20260807 08:43:13.544296 6686 leader_election.cc:288] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:43:13.544375 6689 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f50adcd06148e285ccb354c946995d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 45113 } }
I20260807 08:43:13.544648 6686 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:43:13.544898 6693 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f50adcd06148e285ccb354c946995d. Latest consensus state: current_term: 1 leader_uuid: "e9f50adcd06148e285ccb354c946995d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f50adcd06148e285ccb354c946995d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 45113 } } }
I20260807 08:43:13.544994 6693 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [sys.catalog]: This master's current role is: LEADER
I20260807 08:43:13.545359 6694 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9f50adcd06148e285ccb354c946995d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f50adcd06148e285ccb354c946995d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 45113 } } }
I20260807 08:43:13.545463 6694 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d [sys.catalog]: This master's current role is: LEADER
I20260807 08:43:13.546051 6696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:43:13.546733 6696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:43:13.550379 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:13.544049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.28.108.190:45113)
0807 08:43:13.544415 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:13.544426 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:13.544467 (+ 41us) server_negotiation.cc:407] Connection header received
0807 08:43:13.544609 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:13.544618 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:13.544693 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:13.544868 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:13.545185 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.546693 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.547931 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.548434 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.548588 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:13.549449 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:13.549454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:13.549454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:43:13.549722 (+ 268us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:13.549776 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:43:13.549794 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:13.549880 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:13.550033 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 08:43:13.550133 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":149,"threads_started":1}
I20260807 08:43:13.550370 6696 catalog_manager.cc:1347] Generated new cluster ID: 8510bd73ccc946c785ac8ac7bd0a55b1
I20260807 08:43:13.550530 6696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:43:13.566427 6696 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:43:13.567272 6696 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:43:13.581887 6696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f50adcd06148e285ccb354c946995d: Generated new TSK 0
I20260807 08:43:13.582170 6696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:43:13.587060 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:45113
--builtin_ntp_servers=127.28.108.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 08:43:13.682780 6711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:13.683080 6711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:13.683123 6711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:13.684448 6711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:13.684544 6711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:13.684593 6711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:43:13.686936 6711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:45113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:13.687491 6711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:13.687953 6711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:13.695474 6716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:13.695361 6719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:13.695363 6717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:13.696143 6711 server_base.cc:1034] running on GCE node
I20260807 08:43:13.696429 6711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:13.696832 6711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:13.698058 6711 hybrid_clock.cc:648] HybridClock initialized: now 1786092193698040 us; error 31 us; skew 500 ppm
I20260807 08:43:13.700098 6711 webserver.cc:460] Webserver started at http://127.28.108.129:45509/ using document root <none> and password file <none>
I20260807 08:43:13.700588 6711 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:13.700660 6711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:13.700868 6711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:13.702467 6711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "1bbe701b0efa46faa7cc99257f15348f"
format_stamp: "Formatted at 2026-08-07 08:43:13 on dist-test-slave-9zdj"
server_key: "8a6c6d6b7a09a07a5e6d0c4dfe7d1b7a"
server_key_iv: "12a0e567bcd9b0c2a75fb2399477e434"
server_key_version: "encryptionkey@0"
I20260807 08:43:13.702905 6711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "1bbe701b0efa46faa7cc99257f15348f"
format_stamp: "Formatted at 2026-08-07 08:43:13 on dist-test-slave-9zdj"
server_key: "8a6c6d6b7a09a07a5e6d0c4dfe7d1b7a"
server_key_iv: "12a0e567bcd9b0c2a75fb2399477e434"
server_key_version: "encryptionkey@0"
I20260807 08:43:13.704803 6711 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260807 08:43:13.706115 6725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:13.706545 6711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:13.706707 6711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "1bbe701b0efa46faa7cc99257f15348f"
format_stamp: "Formatted at 2026-08-07 08:43:13 on dist-test-slave-9zdj"
server_key: "8a6c6d6b7a09a07a5e6d0c4dfe7d1b7a"
server_key_iv: "12a0e567bcd9b0c2a75fb2399477e434"
server_key_version: "encryptionkey@0"
I20260807 08:43:13.706944 6711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:13.720611 6711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:13.721079 6711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:13.721267 6711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:13.721560 6711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:43:13.722018 6711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:43:13.722080 6711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:13.722126 6711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:43:13.722150 6711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:13.729259 6711 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:35735
I20260807 08:43:13.729305 6850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:35735 every 8 connection(s)
I20260807 08:43:13.729813 6711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:43:13.734593 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 6711
I20260807 08:43:13.734848 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:43:13.735095 29106 external_mini_cluster.cc:1428] Setting key a046474150238a5074472667d4573150
I20260807 08:43:13.736593 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:13.731040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:49237 (local address 127.28.108.190:45113)
0807 08:43:13.731144 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:13.731147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:13.731454 (+ 307us) server_negotiation.cc:407] Connection header received
0807 08:43:13.731999 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:13.732000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:13.732016 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:13.732073 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:13.732908 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.733463 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.734686 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.734957 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.735051 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:13.736172 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:13.736173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:13.736174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:43:13.736183 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:13.736206 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:43:13.736212 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:13.736301 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:13.736476 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 08:43:13.736521 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 08:43:13.736805 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:45113
--builtin_ntp_servers=127.28.108.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 08:43:13.736920 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:13.730886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:45113 (local address 127.28.108.129:49237)
0807 08:43:13.731222 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:13.731267 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:13.731875 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:13.732152 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:13.732156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:13.732165 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:13.732696 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:13.732701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.733594 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.733595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:13.734556 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:13.734559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.735068 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.735069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:13.735295 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:13.735971 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:43:13.735971 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:13.736008 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:43:13.736038 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:43:13.736315 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:43:13.736325 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:43:13.736328 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:43:13.736428 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 08:43:13.736595 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":128,"threads_started":1}
I20260807 08:43:13.738389 6851 heartbeater.cc:347] Connected to a master server at 127.28.108.190:45113
I20260807 08:43:13.738561 6851 heartbeater.cc:464] Registering TS with master...
I20260807 08:43:13.738863 6851 heartbeater.cc:511] Master 127.28.108.190:45113 requested a full tablet report, sending...
I20260807 08:43:13.739604 6648 ts_manager.cc:194] Registered new tserver with Master: 1bbe701b0efa46faa7cc99257f15348f (127.28.108.129:35735)
I20260807 08:43:13.740356 6648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.129:49237
W20260807 08:43:13.835956 6854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:13.836258 6854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:13.836308 6854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:13.838092 6854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:13.838199 6854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:13.838253 6854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:43:13.839994 6854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:45113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:13.840368 6854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:13.840772 6854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:13.847398 6860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:13.847556 6859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:13.847398 6864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:13.847563 6854 server_base.cc:1034] running on GCE node
I20260807 08:43:13.847904 6854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:13.848202 6854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:13.849428 6854 hybrid_clock.cc:648] HybridClock initialized: now 1786092193849483 us; error 104 us; skew 500 ppm
I20260807 08:43:13.851509 6854 webserver.cc:460] Webserver started at http://127.28.108.130:45685/ using document root <none> and password file <none>
I20260807 08:43:13.851835 6854 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:13.851917 6854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:13.852085 6854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:13.853315 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "e0528e4ba3464f0aaf963370b5608f38"
format_stamp: "Formatted at 2026-08-07 08:43:13 on dist-test-slave-9zdj"
server_key: "bfa0686044f81067dc45f4e660ecef5e"
server_key_iv: "642d9d8c43a82454292dd678d8d265a0"
server_key_version: "encryptionkey@0"
I20260807 08:43:13.853794 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "e0528e4ba3464f0aaf963370b5608f38"
format_stamp: "Formatted at 2026-08-07 08:43:13 on dist-test-slave-9zdj"
server_key: "bfa0686044f81067dc45f4e660ecef5e"
server_key_iv: "642d9d8c43a82454292dd678d8d265a0"
server_key_version: "encryptionkey@0"
I20260807 08:43:13.855525 6854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:13.856626 6871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:13.857118 6854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:13.857420 6854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "e0528e4ba3464f0aaf963370b5608f38"
format_stamp: "Formatted at 2026-08-07 08:43:13 on dist-test-slave-9zdj"
server_key: "bfa0686044f81067dc45f4e660ecef5e"
server_key_iv: "642d9d8c43a82454292dd678d8d265a0"
server_key_version: "encryptionkey@0"
I20260807 08:43:13.857537 6854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:13.891428 6854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:13.891810 6854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:13.891989 6854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:13.892268 6854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:43:13.892664 6854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:43:13.892725 6854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:13.892773 6854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:43:13.892800 6854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:13.899909 6854 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:33817
I20260807 08:43:13.900007 6995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:33817 every 8 connection(s)
I20260807 08:43:13.900454 6854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:43:13.905833 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 6854
I20260807 08:43:13.905973 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:43:13.906216 29106 external_mini_cluster.cc:1428] Setting key 958a424a6ed23a4df66fdecc4ac6c574
I20260807 08:43:13.907717 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:45113
--builtin_ntp_servers=127.28.108.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 08:43:13.907961 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:13.901532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:45005 (local address 127.28.108.190:45113)
0807 08:43:13.901610 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:13.901638 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:13.902086 (+ 448us) server_negotiation.cc:407] Connection header received
0807 08:43:13.902816 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:13.902817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:13.902830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:13.902904 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:13.903888 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.904485 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.905961 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.906205 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.906308 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:13.907519 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:13.907521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:13.907521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:43:13.907527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:13.907548 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:43:13.907554 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:13.907621 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:13.907796 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 08:43:13.907837 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 08:43:13.908301 6998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:13.901505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:45113 (local address 127.28.108.130:45005)
0807 08:43:13.901780 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:13.901956 (+ 176us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:13.902696 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:13.902997 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:13.903001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:13.903011 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:13.903776 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:13.903782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.904596 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.904597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:13.905827 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:13.905836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:13.906336 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:13.906337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:13.906531 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:13.907337 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:43:13.907337 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:13.907382 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:43:13.907411 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:43:13.907679 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:43:13.907687 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:43:13.907690 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:43:13.907770 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 08:43:13.908016 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":140,"threads_started":1}
I20260807 08:43:13.908915 6996 heartbeater.cc:347] Connected to a master server at 127.28.108.190:45113
I20260807 08:43:13.909044 6996 heartbeater.cc:464] Registering TS with master...
I20260807 08:43:13.909273 6996 heartbeater.cc:511] Master 127.28.108.190:45113 requested a full tablet report, sending...
I20260807 08:43:13.909744 6648 ts_manager.cc:194] Registered new tserver with Master: e0528e4ba3464f0aaf963370b5608f38 (127.28.108.130:33817)
I20260807 08:43:13.910156 6648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.130:45005
W20260807 08:43:14.002624 6999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:14.002897 6999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:14.002923 6999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:14.004673 6999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:14.004742 6999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:14.004776 6999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:43:14.006426 6999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:45113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:14.006783 6999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:14.007174 6999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:14.013125 7005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:14.013091 7004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:14.013068 7008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:14.013474 6999 server_base.cc:1034] running on GCE node
I20260807 08:43:14.013774 6999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:14.014019 6999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:14.015188 6999 hybrid_clock.cc:648] HybridClock initialized: now 1786092194015170 us; error 33 us; skew 500 ppm
I20260807 08:43:14.016664 6999 webserver.cc:460] Webserver started at http://127.28.108.131:46127/ using document root <none> and password file <none>
I20260807 08:43:14.017030 6999 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:14.017086 6999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:14.017234 6999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:14.018510 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "74a3a9fa2e8847eebdf890481e3c3f5f"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "ff756e8d2c45b6346268aa439c7aad9d"
server_key_iv: "8f07c431c87a09b48ffcd6a187b6bc0b"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.018903 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "74a3a9fa2e8847eebdf890481e3c3f5f"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "ff756e8d2c45b6346268aa439c7aad9d"
server_key_iv: "8f07c431c87a09b48ffcd6a187b6bc0b"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.020678 6999 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260807 08:43:14.021952 7016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.022289 6999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 08:43:14.022394 6999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "74a3a9fa2e8847eebdf890481e3c3f5f"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "ff756e8d2c45b6346268aa439c7aad9d"
server_key_iv: "8f07c431c87a09b48ffcd6a187b6bc0b"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.022514 6999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:14.036192 6999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:14.036593 6999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:14.036770 6999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:14.037053 6999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:43:14.037462 6999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:43:14.037525 6999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.037562 6999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:43:14.037590 6999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.043985 6999 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:33419
I20260807 08:43:14.044101 7139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:33419 every 8 connection(s)
I20260807 08:43:14.044514 6999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:43:14.045269 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 6999
I20260807 08:43:14.045400 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:43:14.045706 29106 external_mini_cluster.cc:1428] Setting key d55f44a7066f9c1e48428069b65087b7
I20260807 08:43:14.051064 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:14.045501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:46789 (local address 127.28.108.190:45113)
0807 08:43:14.045570 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:14.045572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:14.045953 (+ 381us) server_negotiation.cc:407] Connection header received
0807 08:43:14.046752 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:14.046753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:14.046766 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:14.046814 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:14.047592 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:14.048115 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:14.049197 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:14.049448 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:14.049561 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:14.050665 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:14.050666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:14.050666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:43:14.050672 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:14.050692 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:43:14.050698 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:14.050769 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:14.050921 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 08:43:14.050957 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 08:43:14.051218 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:14.045509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:45113 (local address 127.28.108.131:46789)
0807 08:43:14.045789 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:14.045848 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:14.046637 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:14.046905 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:14.046910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:14.046926 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:14.047480 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:14.047485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:14.048225 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:14.048227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:14.049087 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:14.049090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:14.049567 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:14.049568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:14.049785 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:14.050493 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:43:14.050493 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:14.050528 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:43:14.050554 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:43:14.050787 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:43:14.050792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:43:14.050795 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:43:14.050876 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 08:43:14.050983 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":127,"threads_started":1}
I20260807 08:43:14.052011 7140 heartbeater.cc:347] Connected to a master server at 127.28.108.190:45113
I20260807 08:43:14.052160 7140 heartbeater.cc:464] Registering TS with master...
I20260807 08:43:14.052378 7140 heartbeater.cc:511] Master 127.28.108.190:45113 requested a full tablet report, sending...
I20260807 08:43:14.052874 6648 ts_manager.cc:194] Registered new tserver with Master: 74a3a9fa2e8847eebdf890481e3c3f5f (127.28.108.131:33419)
I20260807 08:43:14.053365 6648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.131:46789
I20260807 08:43:14.057991 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 08:43:14.060133 6692 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:14.059531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34018 (local address 127.28.108.190:45113)
0807 08:43:14.059600 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:14.059603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:14.059677 (+ 74us) server_negotiation.cc:407] Connection header received
0807 08:43:14.059760 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:14.059761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:14.059835 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:14.059904 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:14.060064 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34018: BlockingRecv error: recv got EOF from 127.0.0.1:34018 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 08:43:14.060359 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 6711
I20260807 08:43:14.072777 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 6854
I20260807 08:43:14.079259 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 6999
I20260807 08:43:14.086588 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 6615
2026-08-07T08:43:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (714 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T08:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:43:14Z Disabled control of system clock
I20260807 08:43:14.122613 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:35631
--webserver_interface=127.28.108.190
--webserver_port=0
--builtin_ntp_servers=127.28.108.148:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:35631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 08:43:14.213505 7153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:14.213779 7153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:14.213804 7153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:14.215049 7153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:43:14.215101 7153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:14.215116 7153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:14.215128 7153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:43:14.215142 7153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:43:14.217051 7153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:35631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:35631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:14.217361 7153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:14.217792 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:14.224226 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:14.224349 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:14.224375 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:14.224665 7153 server_base.cc:1034] running on GCE node
I20260807 08:43:14.225023 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:14.225437 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:14.226657 7153 hybrid_clock.cc:648] HybridClock initialized: now 1786092194226629 us; error 47 us; skew 500 ppm
I20260807 08:43:14.228394 7153 webserver.cc:460] Webserver started at http://127.28.108.190:35251/ using document root <none> and password file <none>
I20260807 08:43:14.228783 7153 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:14.228878 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:14.229072 7153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:14.230453 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "2984deaeb55b4afab7ea5ebe71f63685"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "77d2e32685f6dcc9a5b3bea5ba3fd31d"
server_key_iv: "d4abd8a2a60ffd64b5985332d3bc7ba2"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.230901 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "2984deaeb55b4afab7ea5ebe71f63685"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "77d2e32685f6dcc9a5b3bea5ba3fd31d"
server_key_iv: "d4abd8a2a60ffd64b5985332d3bc7ba2"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.232643 7153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:43:14.233819 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.234056 7153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:14.234180 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "2984deaeb55b4afab7ea5ebe71f63685"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "77d2e32685f6dcc9a5b3bea5ba3fd31d"
server_key_iv: "d4abd8a2a60ffd64b5985332d3bc7ba2"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.234283 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:14.238516 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:14.238941 7153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:14.239120 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:14.244242 7153 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:35631
I20260807 08:43:14.244401 7226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:35631 every 8 connection(s)
I20260807 08:43:14.244738 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:43:14.245604 7227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:43:14.249183 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 7153
I20260807 08:43:14.249279 7227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685: Bootstrap starting.
I20260807 08:43:14.249470 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:43:14.249781 29106 external_mini_cluster.cc:1428] Setting key 5df8c90cafdcf6e38f99948f9015f937
I20260807 08:43:14.250253 7227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685: Neither blocks nor log segments found. Creating new log.
I20260807 08:43:14.250730 7227 log.cc:824] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685: Log is configured to *not* fsync() on all Append() calls
I20260807 08:43:14.251823 7227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685: No bootstrap required, opened a new log
I20260807 08:43:14.254032 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:14.250622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.28.108.190:35631)
0807 08:43:14.250918 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:14.250929 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:14.250989 (+ 60us) server_negotiation.cc:407] Connection header received
0807 08:43:14.251858 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:14.251868 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:14.251919 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:14.252168 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:14.253192 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:14.253197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:14.253198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:43:14.253203 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:14.253258 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:43:14.253269 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:14.253573 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:14.253762 (+ 189us) server_negotiation.cc:299] Negotiation successful
0807 08:43:14.253843 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":100,"threads_started":1}
I20260807 08:43:14.254011 7227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984deaeb55b4afab7ea5ebe71f63685" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35631 } }
I20260807 08:43:14.254266 7227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:43:14.254304 7227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2984deaeb55b4afab7ea5ebe71f63685, State: Initialized, Role: FOLLOWER
I20260807 08:43:14.254436 7227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984deaeb55b4afab7ea5ebe71f63685" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35631 } }
I20260807 08:43:14.254606 7227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:43:14.254698 7227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:43:14.254876 7227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:43:14.256109 7227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984deaeb55b4afab7ea5ebe71f63685" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35631 } }
I20260807 08:43:14.256378 7227 leader_election.cc:302] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2984deaeb55b4afab7ea5ebe71f63685; no voters:
I20260807 08:43:14.256613 7227 leader_election.cc:288] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:43:14.256731 7232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:43:14.256942 7232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [term 1 LEADER]: Becoming Leader. State: Replica: 2984deaeb55b4afab7ea5ebe71f63685, State: Running, Role: LEADER
I20260807 08:43:14.256978 7227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:43:14.257087 7232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984deaeb55b4afab7ea5ebe71f63685" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35631 } }
I20260807 08:43:14.258073 7234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2984deaeb55b4afab7ea5ebe71f63685. Latest consensus state: current_term: 1 leader_uuid: "2984deaeb55b4afab7ea5ebe71f63685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984deaeb55b4afab7ea5ebe71f63685" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35631 } } }
I20260807 08:43:14.258204 7234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [sys.catalog]: This master's current role is: LEADER
I20260807 08:43:14.258358 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2984deaeb55b4afab7ea5ebe71f63685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2984deaeb55b4afab7ea5ebe71f63685" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35631 } } }
I20260807 08:43:14.258430 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685 [sys.catalog]: This master's current role is: LEADER
I20260807 08:43:14.258442 7241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:43:14.259006 7241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:43:14.260814 7241 catalog_manager.cc:1347] Generated new cluster ID: 70929ee48c5c4e8883f13591ce753c28
I20260807 08:43:14.260876 7241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:43:14.268667 7241 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:43:14.268954 7241 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:43:14.275653 7241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2984deaeb55b4afab7ea5ebe71f63685: Generated new TSK 0
I20260807 08:43:14.276024 7241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:43:14.280393 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35631
--builtin_ntp_servers=127.28.108.148:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 08:43:14.380400 7252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:14.380643 7252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:14.380669 7252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:14.382119 7252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:14.382179 7252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:14.382210 7252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:43:14.383884 7252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:14.384202 7252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:14.384828 7252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:14.392292 7260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:14.392434 7258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:14.392294 7257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:14.392501 7252 server_base.cc:1034] running on GCE node
I20260807 08:43:14.392992 7252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:14.393337 7252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:14.394521 7252 hybrid_clock.cc:648] HybridClock initialized: now 1786092194394503 us; error 38 us; skew 500 ppm
I20260807 08:43:14.396144 7252 webserver.cc:460] Webserver started at http://127.28.108.129:39913/ using document root <none> and password file <none>
I20260807 08:43:14.396458 7252 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:14.396508 7252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:14.396670 7252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:14.398075 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "e0b3b93f5c6f4dca92e2d2a2159fbbf7"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "dd7f9cad4b41fc5f828c21788ecbf395"
server_key_iv: "586417597d1aefe3b491ca3fbb1906e6"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.398491 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "e0b3b93f5c6f4dca92e2d2a2159fbbf7"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "dd7f9cad4b41fc5f828c21788ecbf395"
server_key_iv: "586417597d1aefe3b491ca3fbb1906e6"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.400053 7252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:43:14.400933 7266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.401129 7252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:14.401283 7252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "e0b3b93f5c6f4dca92e2d2a2159fbbf7"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "dd7f9cad4b41fc5f828c21788ecbf395"
server_key_iv: "586417597d1aefe3b491ca3fbb1906e6"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.401373 7252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:14.405071 7252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:14.405424 7252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:14.405578 7252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:14.405910 7252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:43:14.406368 7252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:43:14.406414 7252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.406438 7252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:43:14.406451 7252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.414319 7252 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:39165
I20260807 08:43:14.414354 7388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:39165 every 8 connection(s)
I20260807 08:43:14.414795 7252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:43:14.418695 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 7252
I20260807 08:43:14.418840 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:43:14.419091 29106 external_mini_cluster.cc:1428] Setting key f755b687616bd675a8a60b52a4e1d9bf
I20260807 08:43:14.419111 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:14.416041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:51319 (local address 127.28.108.190:35631)
0807 08:43:14.416151 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:14.416153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:14.416519 (+ 366us) server_negotiation.cc:407] Connection header received
0807 08:43:14.417057 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:14.417058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:14.417065 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:14.417135 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:14.418637 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:14.418639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:14.418640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:43:14.418646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:14.418667 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:43:14.418683 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:14.418767 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:14.419002 (+ 235us) server_negotiation.cc:299] Negotiation successful
0807 08:43:14.419023 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 08:43:14.419343 7392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:14.416046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35631 (local address 127.28.108.129:51319)
0807 08:43:14.416380 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:14.416421 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:14.416911 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:14.417289 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:14.417293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:14.417303 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:14.418408 (+ 1105us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:43:14.418408 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:14.418444 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:43:14.418480 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:43:14.418865 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:43:14.418876 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:43:14.418881 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:43:14.419013 (+ 132us) client_negotiation.cc:240] Negotiation successful
0807 08:43:14.419096 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":109,"threads_started":1}
I20260807 08:43:14.419972 7390 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35631
I20260807 08:43:14.420097 7390 heartbeater.cc:464] Registering TS with master...
I20260807 08:43:14.420295 7390 heartbeater.cc:511] Master 127.28.108.190:35631 requested a full tablet report, sending...
I20260807 08:43:14.420562 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35631
--builtin_ntp_servers=127.28.108.148:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 08:43:14.420938 7188 ts_manager.cc:194] Registered new tserver with Master: e0b3b93f5c6f4dca92e2d2a2159fbbf7 (127.28.108.129:39165)
W20260807 08:43:14.513931 7393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:14.514256 7393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:14.514303 7393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:14.515903 7393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:14.515988 7393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:14.516036 7393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:43:14.517558 7393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:14.517900 7393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:14.518250 7393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:14.524577 7401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:14.524536 7398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:14.524530 7393 server_base.cc:1034] running on GCE node
W20260807 08:43:14.524498 7399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:14.525357 7393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:14.525758 7393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:14.526947 7393 hybrid_clock.cc:648] HybridClock initialized: now 1786092194526926 us; error 40 us; skew 500 ppm
I20260807 08:43:14.528525 7393 webserver.cc:460] Webserver started at http://127.28.108.130:42363/ using document root <none> and password file <none>
I20260807 08:43:14.528909 7393 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:14.528970 7393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:14.529161 7393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:14.530532 7393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "1e8db827f2b64d95920e236379f72b43"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "a75cf20070ef4e67f1c78f33dd47126b"
server_key_iv: "03de2db0cf9a058e1701eaeb417ccdc3"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.530934 7393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "1e8db827f2b64d95920e236379f72b43"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "a75cf20070ef4e67f1c78f33dd47126b"
server_key_iv: "03de2db0cf9a058e1701eaeb417ccdc3"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.532606 7393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:43:14.534055 7407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.534551 7393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:14.534688 7393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "1e8db827f2b64d95920e236379f72b43"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "a75cf20070ef4e67f1c78f33dd47126b"
server_key_iv: "03de2db0cf9a058e1701eaeb417ccdc3"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.534808 7393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:14.539513 7393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:14.539851 7393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:14.540012 7393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:14.540263 7393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:43:14.540644 7393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:43:14.540685 7393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.540745 7393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:43:14.540778 7393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.548111 7393 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:32881
I20260807 08:43:14.548157 7528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:32881 every 8 connection(s)
I20260807 08:43:14.548604 7393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:43:14.552294 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:14.549541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:46875 (local address 127.28.108.190:35631)
0807 08:43:14.549651 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:14.549654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:14.549957 (+ 303us) server_negotiation.cc:407] Connection header received
0807 08:43:14.550646 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:14.550648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:14.550655 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:14.550713 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:14.552009 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:14.552010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:14.552011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:43:14.552016 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:14.552032 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:43:14.552037 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:14.552098 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:14.552194 (+ 96us) server_negotiation.cc:299] Negotiation successful
0807 08:43:14.552210 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 08:43:14.552515 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:14.549511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35631 (local address 127.28.108.130:46875)
0807 08:43:14.549825 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:14.549863 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:14.550518 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:14.550798 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:14.550802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:14.550811 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:14.551833 (+ 1022us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:43:14.551834 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:14.551866 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:43:14.551899 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:43:14.552099 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:43:14.552106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:43:14.552109 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:43:14.552208 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 08:43:14.552279 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":188,"threads_started":1}
I20260807 08:43:14.553030 7529 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35631
I20260807 08:43:14.553131 7529 heartbeater.cc:464] Registering TS with master...
I20260807 08:43:14.553259 7529 heartbeater.cc:511] Master 127.28.108.190:35631 requested a full tablet report, sending...
I20260807 08:43:14.553728 7188 ts_manager.cc:194] Registered new tserver with Master: 1e8db827f2b64d95920e236379f72b43 (127.28.108.130:32881)
I20260807 08:43:14.557868 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 7393
I20260807 08:43:14.557981 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:43:14.558225 29106 external_mini_cluster.cc:1428] Setting key 8d76d82a5ac5644ddbeda519f76d3841
I20260807 08:43:14.559774 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35631
--builtin_ntp_servers=127.28.108.148:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 08:43:14.651865 7533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:14.652117 7533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:14.652143 7533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:14.653477 7533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:14.653541 7533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:14.653571 7533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:43:14.655160 7533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:14.655501 7533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:14.655865 7533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:14.662277 7539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:14.662408 7533 server_base.cc:1034] running on GCE node
W20260807 08:43:14.662281 7538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:14.662281 7541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:14.662799 7533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:14.663116 7533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:14.664335 7533 hybrid_clock.cc:648] HybridClock initialized: now 1786092194664368 us; error 100 us; skew 500 ppm
I20260807 08:43:14.666133 7533 webserver.cc:460] Webserver started at http://127.28.108.131:46267/ using document root <none> and password file <none>
I20260807 08:43:14.666472 7533 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:14.666575 7533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:14.666749 7533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:14.667941 7533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "f17c68c8ec2f456e8e56c4663363bc8a"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "8c32f10af1293f800823a8102c9018be"
server_key_iv: "fdb5346130ad91b3858d476e021ef615"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.668319 7533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "f17c68c8ec2f456e8e56c4663363bc8a"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "8c32f10af1293f800823a8102c9018be"
server_key_iv: "fdb5346130ad91b3858d476e021ef615"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.669996 7533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260807 08:43:14.670981 7547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.671227 7533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:14.671319 7533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "f17c68c8ec2f456e8e56c4663363bc8a"
format_stamp: "Formatted at 2026-08-07 08:43:14 on dist-test-slave-9zdj"
server_key: "8c32f10af1293f800823a8102c9018be"
server_key_iv: "fdb5346130ad91b3858d476e021ef615"
server_key_version: "encryptionkey@0"
I20260807 08:43:14.671438 7533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:14.675386 7533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:14.675706 7533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:14.675866 7533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:14.676086 7533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:43:14.676419 7533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:43:14.676453 7533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.676509 7533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:43:14.676537 7533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:14.683223 7533 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:42087
I20260807 08:43:14.683287 7667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:42087 every 8 connection(s)
I20260807 08:43:14.683732 7533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:43:14.687031 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 7533
I20260807 08:43:14.687186 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:43:14.687484 29106 external_mini_cluster.cc:1428] Setting key a618db20db0315aa2209823a06ba3294
I20260807 08:43:14.687979 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:14.684849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:47489 (local address 127.28.108.190:35631)
0807 08:43:14.684933 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:14.684936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:14.685162 (+ 226us) server_negotiation.cc:407] Connection header received
0807 08:43:14.685930 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:14.685932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:14.685939 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:14.685999 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:14.687589 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:14.687591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:14.687592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:43:14.687597 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:14.687619 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:43:14.687624 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:14.687777 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:14.687886 (+ 109us) server_negotiation.cc:299] Negotiation successful
0807 08:43:14.687904 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 08:43:14.688267 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:14.684718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35631 (local address 127.28.108.131:47489)
0807 08:43:14.684996 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:14.685041 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:14.685796 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:14.686075 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:14.686080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:14.686090 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:14.687396 (+ 1306us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:43:14.687396 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:14.687431 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:43:14.687475 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:43:14.687778 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:43:14.687786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:43:14.687790 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:43:14.687899 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 08:43:14.687974 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":130,"threads_started":1}
I20260807 08:43:14.688855 7668 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35631
I20260807 08:43:14.688959 7668 heartbeater.cc:464] Registering TS with master...
I20260807 08:43:14.689138 7668 heartbeater.cc:511] Master 127.28.108.190:35631 requested a full tablet report, sending...
I20260807 08:43:14.689649 7188 ts_manager.cc:194] Registered new tserver with Master: f17c68c8ec2f456e8e56c4663363bc8a (127.28.108.131:42087)
I20260807 08:43:14.700224 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 08:43:14.702890 7230 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:43:14.702210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.28.108.190:35631)
0807 08:43:14.702408 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:14.702410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:14.702510 (+ 100us) server_negotiation.cc:407] Connection header received
0807 08:43:14.702587 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:14.702588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:14.702595 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:14.702643 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:14.702804 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55426: BlockingRecv error: recv got EOF from 127.0.0.1:55426 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
I20260807 08:43:14.703003 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 7252
I20260807 08:43:14.711683 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 7393
I20260807 08:43:14.720112 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 7533
I20260807 08:43:14.727579 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 7153
2026-08-07T08:43:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (639 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:43:14 dist-test-slave-9zdj krb5kdc[7678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:43:14 dist-test-slave-9zdj krb5kdc[7678](info): set up 2 sockets
Aug 07 08:43:14 dist-test-slave-9zdj krb5kdc[7678](info): commencing operation
krb5kdc: starting...
W20260807 08:43:15.912513 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:43:16 dist-test-slave-9zdj krb5kdc[7678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:43:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:43:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:43:16.066922 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:40425
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:40425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:43:16.160403 7692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:16.160673 7692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:16.160701 7692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:16.162125 7692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:43:16.162189 7692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:16.162205 7692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:16.162218 7692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:43:16.162231 7692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:43:16.163947 7692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:40425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:40425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:16.164213 7692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:16.164463 7692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:16.170291 7697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:16.170432 7700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:16.170320 7698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:16.170789 7692 server_base.cc:1034] running on GCE node
I20260807 08:43:16.171269 7692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:16.171604 7692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:16.172888 7692 hybrid_clock.cc:648] HybridClock initialized: now 1786092196172870 us; error 30 us; skew 500 ppm
Aug 07 08:43:16 dist-test-slave-9zdj krb5kdc[7678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:43:16.176386 7692 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:43:16.177071 7692 webserver.cc:460] Webserver started at http://127.28.108.190:45913/ using document root <none> and password file <none>
I20260807 08:43:16.177362 7692 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:16.177448 7692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:16.177696 7692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:16.179024 7692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "1abc6396091d41e7b60cfb078289ead9"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "25bc25e327b323a99d4d7376bb2abe52"
server_key_iv: "7e326bc2ea05f343964596f449dc5f0e"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.179486 7692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "1abc6396091d41e7b60cfb078289ead9"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "25bc25e327b323a99d4d7376bb2abe52"
server_key_iv: "7e326bc2ea05f343964596f449dc5f0e"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.181226 7692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 08:43:16.182485 7707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:16.182791 7692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:43:16.182921 7692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "1abc6396091d41e7b60cfb078289ead9"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "25bc25e327b323a99d4d7376bb2abe52"
server_key_iv: "7e326bc2ea05f343964596f449dc5f0e"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.183015 7692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:16.192471 7692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:16.195163 7692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:16.195367 7692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:16.200142 7692 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:40425
I20260807 08:43:16.200182 7766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:40425 every 8 connection(s)
I20260807 08:43:16.200614 7692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:43:16.201894 7767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:43:16.204419 7767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9: Bootstrap starting.
I20260807 08:43:16.204469 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 7692
I20260807 08:43:16.204623 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:43:16.204841 29106 external_mini_cluster.cc:1428] Setting key 0f960fc90d990983b767595c91009478
I20260807 08:43:16.205185 7767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9: Neither blocks nor log segments found. Creating new log.
I20260807 08:43:16.205611 7767 log.cc:824] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9: Log is configured to *not* fsync() on all Append() calls
I20260807 08:43:16.206725 7767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9: No bootstrap required, opened a new log
I20260807 08:43:16.209268 7767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1abc6396091d41e7b60cfb078289ead9" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40425 } }
I20260807 08:43:16.209479 7767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:43:16.209511 7767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc6396091d41e7b60cfb078289ead9, State: Initialized, Role: FOLLOWER
I20260807 08:43:16.209684 7767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1abc6396091d41e7b60cfb078289ead9" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40425 } }
I20260807 08:43:16.209789 7767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:43:16.209861 7767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:43:16.209951 7767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:43:16.210657 7767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1abc6396091d41e7b60cfb078289ead9" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40425 } }
I20260807 08:43:16.210819 7767 leader_election.cc:302] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1abc6396091d41e7b60cfb078289ead9; no voters:
I20260807 08:43:16.211021 7767 leader_election.cc:288] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:43:16.211094 7772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:43:16.211282 7772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [term 1 LEADER]: Becoming Leader. State: Replica: 1abc6396091d41e7b60cfb078289ead9, State: Running, Role: LEADER
I20260807 08:43:16.211360 7767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:43:16.211400 7772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1abc6396091d41e7b60cfb078289ead9" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40425 } }
I20260807 08:43:16.211819 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1abc6396091d41e7b60cfb078289ead9. Latest consensus state: current_term: 1 leader_uuid: "1abc6396091d41e7b60cfb078289ead9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1abc6396091d41e7b60cfb078289ead9" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40425 } } }
I20260807 08:43:16.211952 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [sys.catalog]: This master's current role is: LEADER
I20260807 08:43:16.212168 7780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:43:16.212735 7780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:43:16.213313 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1abc6396091d41e7b60cfb078289ead9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1abc6396091d41e7b60cfb078289ead9" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40425 } } }
I20260807 08:43:16.213385 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9 [sys.catalog]: This master's current role is: LEADER
Aug 07 08:43:16 dist-test-slave-9zdj krb5kdc[7678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:43:16.215109 7780 catalog_manager.cc:1347] Generated new cluster ID: ae4c317f94f049c1a0448efb127643c7
I20260807 08:43:16.215193 7780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:43:16.218644 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.205549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.28.108.190:40425)
0807 08:43:16.205802 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:16.205810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:16.205838 (+ 28us) server_negotiation.cc:407] Connection header received
0807 08:43:16.206456 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:16.206465 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:16.206531 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:16.206663 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:16.207653 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.208657 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.209883 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.210618 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.210730 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.214435 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:16.214439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:16.214440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:43:16.214454 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:16.216682 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:43:16.217184 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:43:16.217188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:43:16.217188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:43:16.217310 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:43:16.217544 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:43:16.217546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:43:16.217546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:43:16.217934 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:16.218113 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:16.218222 (+ 109us) server_negotiation.cc:299] Negotiation successful
0807 08:43:16.218385 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":121,"threads_started":1}
I20260807 08:43:16.235036 7780 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:43:16.235631 7780 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:43:16.243955 7780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9: Generated new TSK 0
I20260807 08:43:16.244199 7780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:43:16.286398 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:40425
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:43:16.377264 7795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:16.377506 7795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:16.377532 7795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:16.378938 7795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:16.378999 7795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:16.379030 7795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:43:16.380555 7795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:40425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:16.380785 7795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:16.380985 7795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:16.386776 7803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:16.386798 7800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:16.386848 7801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:16.387143 7795 server_base.cc:1034] running on GCE node
I20260807 08:43:16.387391 7795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:16.387674 7795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:16.388825 7795 hybrid_clock.cc:648] HybridClock initialized: now 1786092196388806 us; error 29 us; skew 500 ppm
Aug 07 08:43:16 dist-test-slave-9zdj krb5kdc[7678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:43:16.391847 7795 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:43:16.392395 7795 webserver.cc:460] Webserver started at http://127.28.108.129:46263/ using document root <none> and password file <none>
I20260807 08:43:16.392681 7795 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:16.392762 7795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:16.392939 7795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:16.394201 7795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "7dc8574703674c17b898ecda0001ab47"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "265d532db68b33f1f7dc8ea36d6a9b43"
server_key_iv: "af24c04865ffaaa2c0388d7de88c7831"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.394599 7795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "7dc8574703674c17b898ecda0001ab47"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "265d532db68b33f1f7dc8ea36d6a9b43"
server_key_iv: "af24c04865ffaaa2c0388d7de88c7831"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.396171 7795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:43:16.397166 7810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:16.397430 7795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:16.397555 7795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "7dc8574703674c17b898ecda0001ab47"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "265d532db68b33f1f7dc8ea36d6a9b43"
server_key_iv: "af24c04865ffaaa2c0388d7de88c7831"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.397682 7795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:16.417989 7795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:16.420815 7795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:16.421036 7795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:16.421470 7795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:43:16.421958 7795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:43:16.422034 7795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:16.422077 7795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:43:16.422106 7795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:16.429072 7795 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:42573
I20260807 08:43:16.429124 7939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:42573 every 8 connection(s)
I20260807 08:43:16.429533 7795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:43:16.434048 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 7795
I20260807 08:43:16.434209 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:43:16.434459 29106 external_mini_cluster.cc:1428] Setting key 0c7779079ca119dbddf6a4894740b169
Aug 07 08:43:16 dist-test-slave-9zdj krb5kdc[7678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:43:16.441462 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.430432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:36789 (local address 127.28.108.190:40425)
0807 08:43:16.430512 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:16.430514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:16.430792 (+ 278us) server_negotiation.cc:407] Connection header received
0807 08:43:16.431368 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:16.431370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:16.431384 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:16.431434 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:16.432571 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.433134 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.434286 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.434632 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.434717 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.437844 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:16.437847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:16.437847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:43:16.437860 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:16.439410 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:43:16.439961 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:43:16.439962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:43:16.439963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:43:16.440045 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:43:16.440346 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:43:16.440347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:43:16.440348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:43:16.440525 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:16.440616 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:16.441184 (+ 568us) server_negotiation.cc:299] Negotiation successful
0807 08:43:16.441314 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 08:43:16.441509 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.430436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:40425 (local address 127.28.108.129:36789)
0807 08:43:16.430635 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:16.430677 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:16.431241 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:16.431513 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:16.431518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:16.432168 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:16.432428 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.432433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.433262 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.433263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.434160 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.434163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.434797 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.434798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.434973 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.435758 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:16.435768 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:16.437723 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:43:16.439544 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:43:16.439556 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:43:16.439558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:43:16.439829 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:43:16.440131 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:43:16.440132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:43:16.440132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:43:16.440265 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:43:16.440641 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:43:16.440650 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:43:16.440948 (+ 298us) client_negotiation.cc:769] Sending connection context
0807 08:43:16.441115 (+ 167us) client_negotiation.cc:240] Negotiation successful
0807 08:43:16.441291 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260807 08:43:16.442485 7940 heartbeater.cc:347] Connected to a master server at 127.28.108.190:40425
I20260807 08:43:16.442628 7940 heartbeater.cc:464] Registering TS with master...
I20260807 08:43:16.442866 7940 heartbeater.cc:511] Master 127.28.108.190:40425 requested a full tablet report, sending...
I20260807 08:43:16.443758 7728 ts_manager.cc:194] Registered new tserver with Master: 7dc8574703674c17b898ecda0001ab47 (127.28.108.129:42573)
I20260807 08:43:16.444802 7728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:36789
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:43:16.475505 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:40425
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:43:16.569664 7945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:16.569940 7945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:16.569972 7945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:16.571527 7945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:16.571612 7945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:16.571650 7945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:43:16.573664 7945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:40425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:16.574031 7945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:16.574373 7945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:16.581076 7953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:16.581367 7952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:16.581378 7955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:16.581578 7945 server_base.cc:1034] running on GCE node
I20260807 08:43:16.581990 7945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:16.582350 7945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:16.583557 7945 hybrid_clock.cc:648] HybridClock initialized: now 1786092196583494 us; error 80 us; skew 500 ppm
Aug 07 08:43:16 dist-test-slave-9zdj krb5kdc[7678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:43:16.587424 7945 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:43:16.588049 7945 webserver.cc:460] Webserver started at http://127.28.108.130:45921/ using document root <none> and password file <none>
I20260807 08:43:16.588286 7945 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:16.588333 7945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:16.588435 7945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:16.589541 7945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "71a81cd2dd024df5906c9a15af12ef3f"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "6d13d9fb4e73870cd256eced9c9acafc"
server_key_iv: "e7bf25ce484a7e53069d5d715e9e0e43"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.589980 7945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "71a81cd2dd024df5906c9a15af12ef3f"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "6d13d9fb4e73870cd256eced9c9acafc"
server_key_iv: "e7bf25ce484a7e53069d5d715e9e0e43"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.591486 7945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:16.592370 7965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:16.592669 7945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:16.592804 7945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "71a81cd2dd024df5906c9a15af12ef3f"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "6d13d9fb4e73870cd256eced9c9acafc"
server_key_iv: "e7bf25ce484a7e53069d5d715e9e0e43"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.592912 7945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:16.618201 7945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:16.620836 7945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:16.621031 7945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:16.621311 7945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:43:16.621678 7945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:43:16.621744 7945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:16.621778 7945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:43:16.621796 7945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:16.628412 7945 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:42573
I20260807 08:43:16.628517 8086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:42573 every 8 connection(s)
I20260807 08:43:16.628880 7945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:43:16.633478 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 7945
I20260807 08:43:16.633598 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:43:16.633947 29106 external_mini_cluster.cc:1428] Setting key 4739f3d16459ad26f87cc6c7b6b0e0d6
Aug 07 08:43:16 dist-test-slave-9zdj krb5kdc[7678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:43:16.640477 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.629887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:46421 (local address 127.28.108.190:40425)
0807 08:43:16.629981 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:16.629984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:16.630189 (+ 205us) server_negotiation.cc:407] Connection header received
0807 08:43:16.630727 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:16.630729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:16.630742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:16.630809 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:16.631869 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.632443 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.633838 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.634092 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.634181 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.636962 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:16.636964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:16.636964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:43:16.636972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:16.638793 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:43:16.639272 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:43:16.639273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:43:16.639274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:43:16.639339 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:43:16.639617 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:43:16.639628 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:43:16.639629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:43:16.639768 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:16.639852 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:16.640289 (+ 437us) server_negotiation.cc:299] Negotiation successful
0807 08:43:16.640382 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 08:43:16.640592 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.629850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:40425 (local address 127.28.108.130:46421)
0807 08:43:16.630041 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:16.630082 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:16.630614 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:16.630869 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:16.630872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:16.631509 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:16.631733 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.631739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.632566 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.632567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.633710 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.633714 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.634196 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.634197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.634379 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.635105 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:16.635114 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:16.636819 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:43:16.638918 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:43:16.638926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:43:16.638929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:43:16.639187 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:43:16.639413 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:43:16.639414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:43:16.639414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:43:16.639539 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:43:16.639842 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:43:16.639850 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:43:16.640014 (+ 164us) client_negotiation.cc:769] Sending connection context
0807 08:43:16.640129 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 08:43:16.640335 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":120,"threads_started":1}
I20260807 08:43:16.641075 8087 heartbeater.cc:347] Connected to a master server at 127.28.108.190:40425
I20260807 08:43:16.641422 8087 heartbeater.cc:464] Registering TS with master...
I20260807 08:43:16.641840 8087 heartbeater.cc:511] Master 127.28.108.190:40425 requested a full tablet report, sending...
I20260807 08:43:16.642354 7728 ts_manager.cc:194] Registered new tserver with Master: 71a81cd2dd024df5906c9a15af12ef3f (127.28.108.130:42573)
I20260807 08:43:16.642817 7728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:46421
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:43:16.670497 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:40425
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:43:16.768539 8093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:43:16.768791 8093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:43:16.768819 8093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:43:16.770128 8093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:43:16.770182 8093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:43:16.770215 8093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:43:16.771780 8093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:40425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:43:16.772006 8093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:43:16.772212 8093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:43:16.778127 8100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:16.778177 8098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:43:16.778308 8103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:43:16.778591 8093 server_base.cc:1034] running on GCE node
I20260807 08:43:16.778889 8093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:43:16.779342 8093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:43:16.780561 8093 hybrid_clock.cc:648] HybridClock initialized: now 1786092196780545 us; error 30 us; skew 500 ppm
Aug 07 08:43:16 dist-test-slave-9zdj krb5kdc[7678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:43:16.783931 8093 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:43:16.784617 8093 webserver.cc:460] Webserver started at http://127.28.108.131:37865/ using document root <none> and password file <none>
I20260807 08:43:16.784881 8093 fs_manager.cc:359] Metadata directory not provided
I20260807 08:43:16.784961 8093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:43:16.785135 8093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:43:16.786450 8093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "67beb1994d9144b4a64457eb1378b4cf"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "6b1ec40db2163bb6b0af8ae118caf8bc"
server_key_iv: "85c48ae58003659a0b58ac8e328220ed"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.786936 8093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "67beb1994d9144b4a64457eb1378b4cf"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "6b1ec40db2163bb6b0af8ae118caf8bc"
server_key_iv: "85c48ae58003659a0b58ac8e328220ed"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.788690 8093 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260807 08:43:16.789767 8111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:16.790246 8093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:43:16.790385 8093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "67beb1994d9144b4a64457eb1378b4cf"
format_stamp: "Formatted at 2026-08-07 08:43:16 on dist-test-slave-9zdj"
server_key: "6b1ec40db2163bb6b0af8ae118caf8bc"
server_key_iv: "85c48ae58003659a0b58ac8e328220ed"
server_key_version: "encryptionkey@0"
I20260807 08:43:16.790488 8093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:43:16.801469 8093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:43:16.804230 8093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:43:16.804455 8093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:43:16.804744 8093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:43:16.805145 8093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:43:16.805316 8093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:16.805366 8093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:43:16.805390 8093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:43:16.812558 8093 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:38029
I20260807 08:43:16.812677 8231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:38029 every 8 connection(s)
I20260807 08:43:16.813074 8093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:43:16.817950 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 8093
I20260807 08:43:16.818114 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:43:16.818327 29106 external_mini_cluster.cc:1428] Setting key 4134ee27983c119c9a85a0cb32e0d296
Aug 07 08:43:16 dist-test-slave-9zdj krb5kdc[7678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092196, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:43:16.824343 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.814049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:54379 (local address 127.28.108.190:40425)
0807 08:43:16.814097 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:16.814099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:16.814312 (+ 213us) server_negotiation.cc:407] Connection header received
0807 08:43:16.814926 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:16.814927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:16.814942 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:16.814996 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:16.816062 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.816597 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.817704 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.817917 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.818014 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.820917 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:16.820918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:16.820919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:43:16.820927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:16.822797 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:43:16.823229 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:43:16.823230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:43:16.823231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:43:16.823290 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:43:16.823611 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:43:16.823612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:43:16.823612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:43:16.823779 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:16.823840 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:16.824173 (+ 333us) server_negotiation.cc:299] Negotiation successful
0807 08:43:16.824274 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 08:43:16.824575 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.813968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:40425 (local address 127.28.108.131:54379)
0807 08:43:16.814163 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:16.814210 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:16.814750 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:16.815077 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:16.815082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:16.815723 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:43:16.815949 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.815954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.816715 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.816716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.817562 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.817564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.818024 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.818025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.818223 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.818882 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:43:16.818894 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:43:16.820795 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:43:16.822894 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:43:16.822901 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:43:16.822903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:43:16.823151 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:43:16.823393 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:43:16.823394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:43:16.823394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:43:16.823528 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:43:16.823864 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:43:16.823871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:43:16.824013 (+ 142us) client_negotiation.cc:769] Sending connection context
0807 08:43:16.824111 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 08:43:16.824249 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":105,"threads_started":1}
I20260807 08:43:16.825173 8232 heartbeater.cc:347] Connected to a master server at 127.28.108.190:40425
I20260807 08:43:16.825366 8232 heartbeater.cc:464] Registering TS with master...
I20260807 08:43:16.825603 8232 heartbeater.cc:511] Master 127.28.108.190:40425 requested a full tablet report, sending...
I20260807 08:43:16.826198 7728 ts_manager.cc:194] Registered new tserver with Master: 67beb1994d9144b4a64457eb1378b4cf (127.28.108.131:38029)
I20260807 08:43:16.826671 7728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:54379
I20260807 08:43:16.830467 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:43:16.839107 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.832091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.28.108.190:40425)
0807 08:43:16.832162 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:16.832164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:16.832244 (+ 80us) server_negotiation.cc:407] Connection header received
0807 08:43:16.832392 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:16.832393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:16.832403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:16.832464 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:43:16.833069 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.833783 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.834541 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.834743 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.834815 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.836184 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:43:16.836186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:43:16.836186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:43:16.836193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:43:16.837798 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:43:16.838174 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:43:16.838176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:43:16.838176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:43:16.838252 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:43:16.838510 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:43:16.838511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:43:16.838511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:43:16.838758 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:43:16.838852 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:16.838951 (+ 99us) server_negotiation.cc:299] Negotiation successful
0807 08:43:16.839027 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 08:43:16.841341 7728 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:43:16.843183 7728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:43:16.851354 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.846830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.28.108.130:42573)
0807 08:43:16.847062 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:16.847066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:16.847077 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:43:16.847116 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:16.847119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:16.847137 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:16.847228 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:43:16.847502 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.848429 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.850112 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.850857 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.850959 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.851028 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:16.851197 (+ 169us) server_negotiation.cc:299] Negotiation successful
0807 08:43:16.851248 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":103,"threads_started":1}
I20260807 08:43:16.851398 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.846576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:42573 (local address 127.0.0.1:49964)
0807 08:43:16.846958 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:16.846969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:16.847019 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:16.847322 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:16.847323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:16.847326 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:43:16.847391 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.847393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.848577 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.848578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.849991 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.849994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.850981 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.850982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.851061 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.851076 (+ 15us) client_negotiation.cc:769] Sending connection context
0807 08:43:16.851161 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 08:43:16.851212 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":5,"thread_start_us":88,"threads_started":1}
I20260807 08:43:16.852017 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.846571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.28.108.129:42573)
0807 08:43:16.846787 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:16.846792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:16.846996 (+ 204us) server_negotiation.cc:407] Connection header received
0807 08:43:16.847144 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:16.847150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:16.847172 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:16.847249 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:43:16.847505 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.848848 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.850579 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.851473 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.851564 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.851640 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:16.851878 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 08:43:16.851926 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
I20260807 08:43:16.851984 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.846320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:42573 (local address 127.0.0.1:39300)
0807 08:43:16.846842 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:16.846855 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:16.846925 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:16.847332 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:16.847332 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:16.847334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:43:16.847405 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.847406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.848998 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.848999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.850431 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.850434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.851686 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.851688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.851777 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.851783 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:43:16.851849 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 08:43:16.851902 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":53,"thread_start_us":134,"threads_started":1}
I20260807 08:43:16.853055 7869 tablet_service.cc:1467] Processing CreateTablet for tablet c3edf21ae46d4ba794ff833ecfbf440f (DEFAULT_TABLE table=test-table [id=a0ca730e1a72447792bf100ee09002ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:43:16.853055 8016 tablet_service.cc:1467] Processing CreateTablet for tablet c3edf21ae46d4ba794ff833ecfbf440f (DEFAULT_TABLE table=test-table [id=a0ca730e1a72447792bf100ee09002ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:43:16.853561 8016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3edf21ae46d4ba794ff833ecfbf440f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:43:16.853880 7869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3edf21ae46d4ba794ff833ecfbf440f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:43:16.857288 8253 tablet_bootstrap.cc:492] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: Bootstrap starting.
I20260807 08:43:16.857345 8254 tablet_bootstrap.cc:492] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47: Bootstrap starting.
I20260807 08:43:16.858170 8254 tablet_bootstrap.cc:654] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47: Neither blocks nor log segments found. Creating new log.
I20260807 08:43:16.858186 8253 tablet_bootstrap.cc:654] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: Neither blocks nor log segments found. Creating new log.
I20260807 08:43:16.858924 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.846275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:38029 (local address 127.0.0.1:33496)
0807 08:43:16.846515 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:16.846568 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:16.846650 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:16.847093 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:16.847096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:16.847098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:43:16.847227 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.847229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.848985 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.848987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.850433 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:16.850435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.858506 (+ 8071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.858510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:16.858635 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.858644 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 08:43:16.858735 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 08:43:16.858818 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":45,"thread_start_us":69,"threads_started":1}
I20260807 08:43:16.858896 8254 log.cc:824] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47: Log is configured to *not* fsync() on all Append() calls
I20260807 08:43:16.859241 8253 log.cc:824] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: Log is configured to *not* fsync() on all Append() calls
I20260807 08:43:16.859289 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:16.846373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.28.108.131:38029)
0807 08:43:16.846630 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:16.846636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:16.846649 (+ 13us) server_negotiation.cc:407] Connection header received
0807 08:43:16.846793 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:16.846796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:16.846815 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:16.846897 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:43:16.847525 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.848839 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.852074 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:16.858372 (+ 6298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:16.858480 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:16.858556 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:16.859127 (+ 571us) server_negotiation.cc:299] Negotiation successful
0807 08:43:16.859196 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":114,"threads_started":1}
I20260807 08:43:16.859974 8254 tablet_bootstrap.cc:492] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47: No bootstrap required, opened a new log
I20260807 08:43:16.860066 8254 ts_tablet_manager.cc:1397] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:43:16.860067 8164 tablet_service.cc:1467] Processing CreateTablet for tablet c3edf21ae46d4ba794ff833ecfbf440f (DEFAULT_TABLE table=test-table [id=a0ca730e1a72447792bf100ee09002ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:43:16.860460 8164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3edf21ae46d4ba794ff833ecfbf440f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:43:16.862516 8254 raft_consensus.cc:348] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67beb1994d9144b4a64457eb1378b4cf" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 38029 } } peers { permanent_uuid: "7dc8574703674c17b898ecda0001ab47" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 42573 } } peers { permanent_uuid: "71a81cd2dd024df5906c9a15af12ef3f" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42573 } }
I20260807 08:43:16.862744 8254 raft_consensus.cc:374] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:43:16.862788 8254 raft_consensus.cc:729] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc8574703674c17b898ecda0001ab47, State: Initialized, Role: FOLLOWER
I20260807 08:43:16.862967 8254 consensus_queue.cc:260] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67beb1994d9144b4a64457eb1378b4cf" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 38029 } } peers { permanent_uuid: "7dc8574703674c17b898ecda0001ab47" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 42573 } } peers { permanent_uuid: "71a81cd2dd024df5906c9a15af12ef3f" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42573 } }
I20260807 08:43:16.863350 8254 ts_tablet_manager.cc:1428] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 08:43:16.863387 7940 heartbeater.cc:503] Master 127.28.108.190:40425 was elected leader, sending a full tablet report...
I20260807 08:43:16.864301 8258 tablet_bootstrap.cc:492] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: Bootstrap starting.
I20260807 08:43:16.864626 8253 tablet_bootstrap.cc:492] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: No bootstrap required, opened a new log
I20260807 08:43:16.864727 8253 ts_tablet_manager.cc:1397] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260807 08:43:16.865419 8258 tablet_bootstrap.cc:654] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: Neither blocks nor log segments found. Creating new log.
I20260807 08:43:16.866164 8258 log.cc:824] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: Log is configured to *not* fsync() on all Append() calls
I20260807 08:43:16.867157 8258 tablet_bootstrap.cc:492] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: No bootstrap required, opened a new log
I20260807 08:43:16.867245 8258 ts_tablet_manager.cc:1397] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260807 08:43:16.867393 8253 raft_consensus.cc:348] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67beb1994d9144b4a64457eb1378b4cf" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 38029 } } peers { permanent_uuid: "7dc8574703674c17b898ecda0001ab47" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 42573 } } peers { permanent_uuid: "71a81cd2dd024df5906c9a15af12ef3f" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42573 } }
I20260807 08:43:16.867594 8253 raft_consensus.cc:374] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:43:16.867636 8253 raft_consensus.cc:729] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a81cd2dd024df5906c9a15af12ef3f, State: Initialized, Role: FOLLOWER
I20260807 08:43:16.867781 8253 consensus_queue.cc:260] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67beb1994d9144b4a64457eb1378b4cf" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 38029 } } peers { permanent_uuid: "7dc8574703674c17b898ecda0001ab47" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 42573 } } peers { permanent_uuid: "71a81cd2dd024df5906c9a15af12ef3f" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42573 } }
I20260807 08:43:16.868124 8253 ts_tablet_manager.cc:1428] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 08:43:16.868283 8087 heartbeater.cc:503] Master 127.28.108.190:40425 was elected leader, sending a full tablet report...
I20260807 08:43:16.869496 8258 raft_consensus.cc:348] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67beb1994d9144b4a64457eb1378b4cf" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 38029 } } peers { permanent_uuid: "7dc8574703674c17b898ecda0001ab47" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 42573 } } peers { permanent_uuid: "71a81cd2dd024df5906c9a15af12ef3f" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42573 } }
I20260807 08:43:16.869730 8258 raft_consensus.cc:374] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:43:16.869801 8258 raft_consensus.cc:729] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67beb1994d9144b4a64457eb1378b4cf, State: Initialized, Role: FOLLOWER
I20260807 08:43:16.869940 8258 consensus_queue.cc:260] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67beb1994d9144b4a64457eb1378b4cf" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 38029 } } peers { permanent_uuid: "7dc8574703674c17b898ecda0001ab47" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 42573 } } peers { permanent_uuid: "71a81cd2dd024df5906c9a15af12ef3f" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42573 } }
I20260807 08:43:16.870312 8232 heartbeater.cc:503] Master 127.28.108.190:40425 was elected leader, sending a full tablet report...
I20260807 08:43:16.870306 8258 ts_tablet_manager.cc:1428] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260807 08:43:16.879606 8089 tablet.cc:2369] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:43:16.930476 7941 tablet.cc:2369] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:43:17.063870 8234 tablet.cc:2369] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:43:17.095000 8257 raft_consensus.cc:482] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:43:17.095256 8257 raft_consensus.cc:504] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67beb1994d9144b4a64457eb1378b4cf" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 38029 } } peers { permanent_uuid: "7dc8574703674c17b898ecda0001ab47" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 42573 } } peers { permanent_uuid: "71a81cd2dd024df5906c9a15af12ef3f" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42573 } }
I20260807 08:43:17.095742 8257 leader_election.cc:288] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67beb1994d9144b4a64457eb1378b4cf (127.28.108.131:38029), 71a81cd2dd024df5906c9a15af12ef3f (127.28.108.130:42573)
I20260807 08:43:17.100684 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:17.096029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:38029 (local address 127.28.108.129:32777)
0807 08:43:17.096193 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:17.096206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:17.096302 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:17.096483 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:17.096484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:17.096487 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:43:17.096582 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:17.096585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:17.097661 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:17.097662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:17.099158 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:17.099162 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:17.100368 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:17.100369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:17.100437 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:17.100440 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:43:17.100545 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 08:43:17.100597 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":53,"thread_start_us":77,"threads_started":1}
I20260807 08:43:17.100665 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:17.096218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:50265 (local address 127.28.108.130:42573)
0807 08:43:17.096325 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:17.096327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:17.096336 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:43:17.096370 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:17.096371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:17.096384 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:17.096452 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:43:17.096752 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:17.097582 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:17.099375 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:17.100022 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:17.100136 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:17.100150 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:17.100516 (+ 366us) server_negotiation.cc:299] Negotiation successful
0807 08:43:17.100578 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 08:43:17.100843 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:17.096103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:32777 (local address 127.28.108.131:38029)
0807 08:43:17.096164 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:17.096167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:17.096308 (+ 141us) server_negotiation.cc:407] Connection header received
0807 08:43:17.096369 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:17.096370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:17.096382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:17.096452 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:43:17.096752 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:17.097535 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:17.099376 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:17.100101 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:17.100182 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:17.100190 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:17.100652 (+ 462us) server_negotiation.cc:299] Negotiation successful
0807 08:43:17.100708 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260807 08:43:17.100970 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:17.096098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:42573 (local address 127.28.108.129:50265)
0807 08:43:17.096234 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:17.096244 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:43:17.096302 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:43:17.096493 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:43:17.096494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:43:17.096495 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:43:17.096590 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:17.096591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:17.097716 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:17.097716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:17.099159 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:43:17.099162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:43:17.100303 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:43:17.100304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:43:17.100398 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:17.100405 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 08:43:17.100837 (+ 432us) client_negotiation.cc:240] Negotiation successful
0807 08:43:17.100892 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":34,"thread_start_us":89,"threads_started":1}
I20260807 08:43:17.101521 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3edf21ae46d4ba794ff833ecfbf440f" candidate_uuid: "7dc8574703674c17b898ecda0001ab47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67beb1994d9144b4a64457eb1378b4cf" is_pre_election: true
I20260807 08:43:17.101773 8185 raft_consensus.cc:2454] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dc8574703674c17b898ecda0001ab47 in term 0.
I20260807 08:43:17.102010 8037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3edf21ae46d4ba794ff833ecfbf440f" candidate_uuid: "7dc8574703674c17b898ecda0001ab47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a81cd2dd024df5906c9a15af12ef3f" is_pre_election: true
I20260807 08:43:17.102104 7812 leader_election.cc:302] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67beb1994d9144b4a64457eb1378b4cf, 7dc8574703674c17b898ecda0001ab47; no voters:
I20260807 08:43:17.102245 8037 raft_consensus.cc:2454] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dc8574703674c17b898ecda0001ab47 in term 0.
I20260807 08:43:17.102276 8257 raft_consensus.cc:2781] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:43:17.102351 8257 raft_consensus.cc:482] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:43:17.102372 8257 raft_consensus.cc:3037] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:43:17.103257 8257 raft_consensus.cc:504] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67beb1994d9144b4a64457eb1378b4cf" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 38029 } } peers { permanent_uuid: "7dc8574703674c17b898ecda0001ab47" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 42573 } } peers { permanent_uuid: "71a81cd2dd024df5906c9a15af12ef3f" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42573 } }
I20260807 08:43:17.103454 8257 leader_election.cc:288] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [CANDIDATE]: Term 1 election: Requested vote from peers 67beb1994d9144b4a64457eb1378b4cf (127.28.108.131:38029), 71a81cd2dd024df5906c9a15af12ef3f (127.28.108.130:42573)
I20260807 08:43:17.103663 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3edf21ae46d4ba794ff833ecfbf440f" candidate_uuid: "7dc8574703674c17b898ecda0001ab47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67beb1994d9144b4a64457eb1378b4cf"
I20260807 08:43:17.103762 8185 raft_consensus.cc:3037] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:43:17.103785 8037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3edf21ae46d4ba794ff833ecfbf440f" candidate_uuid: "7dc8574703674c17b898ecda0001ab47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a81cd2dd024df5906c9a15af12ef3f"
I20260807 08:43:17.103919 8037 raft_consensus.cc:3037] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:43:17.104720 8185 raft_consensus.cc:2454] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dc8574703674c17b898ecda0001ab47 in term 1.
I20260807 08:43:17.104854 8037 raft_consensus.cc:2454] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7dc8574703674c17b898ecda0001ab47 in term 1.
I20260807 08:43:17.104961 7812 leader_election.cc:302] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67beb1994d9144b4a64457eb1378b4cf, 7dc8574703674c17b898ecda0001ab47; no voters:
I20260807 08:43:17.105136 8257 raft_consensus.cc:2781] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:43:17.105334 8257 raft_consensus.cc:686] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [term 1 LEADER]: Becoming Leader. State: Replica: 7dc8574703674c17b898ecda0001ab47, State: Running, Role: LEADER
I20260807 08:43:17.105430 8257 consensus_queue.cc:237] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67beb1994d9144b4a64457eb1378b4cf" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 38029 } } peers { permanent_uuid: "7dc8574703674c17b898ecda0001ab47" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 42573 } } peers { permanent_uuid: "71a81cd2dd024df5906c9a15af12ef3f" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42573 } }
I20260807 08:43:17.106552 7727 catalog_manager.cc:5495] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7dc8574703674c17b898ecda0001ab47 (127.28.108.129). New cstate: current_term: 1 leader_uuid: "7dc8574703674c17b898ecda0001ab47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67beb1994d9144b4a64457eb1378b4cf" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 38029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7dc8574703674c17b898ecda0001ab47" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 42573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a81cd2dd024df5906c9a15af12ef3f" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42573 } health_report { overall_health: UNKNOWN } } }
I20260807 08:43:17.139818 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:17.136651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.28.108.129:42573)
0807 08:43:17.136737 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0807 08:43:17.136739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:43:17.136749 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:43:17.136797 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:43:17.136798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:43:17.136824 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:43:17.136905 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:43:17.137108 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:17.137719 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:17.138693 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:43:17.138967 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:43:17.139041 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:43:17.139362 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:43:17.139496 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:43:17.139576 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 08:43:17.139676 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 08:43:17.139748 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 08:43:17.142256 8037 raft_consensus.cc:1261] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f [term 1 FOLLOWER]: Refusing update from remote peer 7dc8574703674c17b898ecda0001ab47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:43:17.142256 8185 raft_consensus.cc:1261] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf [term 1 FOLLOWER]: Refusing update from remote peer 7dc8574703674c17b898ecda0001ab47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:43:17.142745 8257 consensus_queue.cc:1035] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a81cd2dd024df5906c9a15af12ef3f" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:43:17.142936 8264 consensus_queue.cc:1035] T c3edf21ae46d4ba794ff833ecfbf440f P 7dc8574703674c17b898ecda0001ab47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67beb1994d9144b4a64457eb1378b4cf" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 38029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:43:17.146065 8268 mvcc.cc:205] Tried to move back new op lower bound from 7315833639490482176 to 7315833639344812032. Current Snapshot: MvccSnapshot[applied={T|T < 7315833639490482176}]
I20260807 08:43:17.148854 7728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:43:17.148978 7728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:43:17.149987 7728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1abc6396091d41e7b60cfb078289ead9: Sending DeleteTablet for 3 replicas of tablet c3edf21ae46d4ba794ff833ecfbf440f
I20260807 08:43:17.150394 8164 tablet_service.cc:1514] Processing DeleteTablet for tablet c3edf21ae46d4ba794ff833ecfbf440f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:43:17 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:33496
I20260807 08:43:17.150439 8016 tablet_service.cc:1514] Processing DeleteTablet for tablet c3edf21ae46d4ba794ff833ecfbf440f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:43:17 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:49964
I20260807 08:43:17.150741 7869 tablet_service.cc:1514] Processing DeleteTablet for tablet c3edf21ae46d4ba794ff833ecfbf440f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:43:17 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:39300
I20260807 08:43:17.150797 8279 tablet_replica.cc:330] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: stopping tablet replica
I20260807 08:43:17.150909 8279 raft_consensus.cc:2229] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:43:17.151029 8279 raft_consensus.cc:2258] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:43:17.151010 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 7795
I20260807 08:43:17.151206 8278 tablet_replica.cc:330] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: stopping tablet replica
I20260807 08:43:17.151321 8278 raft_consensus.cc:2229] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:43:17.151417 8278 raft_consensus.cc:2258] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:43:17.152781 8279 ts_tablet_manager.cc:1905] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:43:17.154501 8278 ts_tablet_manager.cc:1905] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:43:17.154711 8279 ts_tablet_manager.cc:1918] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:43:17.154791 8279 log.cc:1196] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/c3edf21ae46d4ba794ff833ecfbf440f
I20260807 08:43:17.155104 8279 ts_tablet_manager.cc:1939] T c3edf21ae46d4ba794ff833ecfbf440f P 71a81cd2dd024df5906c9a15af12ef3f: Deleting consensus metadata
I20260807 08:43:17.155598 7708 catalog_manager.cc:4841] TS 71a81cd2dd024df5906c9a15af12ef3f (127.28.108.130:42573): tablet c3edf21ae46d4ba794ff833ecfbf440f (table test-table [id=a0ca730e1a72447792bf100ee09002ad]) successfully deleted
I20260807 08:43:17.156759 8278 ts_tablet_manager.cc:1918] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:43:17.156877 8278 log.cc:1196] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/c3edf21ae46d4ba794ff833ecfbf440f
I20260807 08:43:17.157191 8278 ts_tablet_manager.cc:1939] T c3edf21ae46d4ba794ff833ecfbf440f P 67beb1994d9144b4a64457eb1378b4cf: Deleting consensus metadata
I20260807 08:43:17.157780 7710 catalog_manager.cc:4841] TS 67beb1994d9144b4a64457eb1378b4cf (127.28.108.131:38029): tablet c3edf21ae46d4ba794ff833ecfbf440f (table test-table [id=a0ca730e1a72447792bf100ee09002ad]) successfully deleted
W20260807 08:43:17.161566 7708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:42573 (error 108)
I20260807 08:43:17.161935 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 7945
I20260807 08:43:17.164053 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:43:17.162151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:42573 (local address 127.0.0.1:39316)
0807 08:43:17.163927 (+ 1776us) negotiation.cc:107] Waiting for socket to connect
0807 08:43:17.164000 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:42573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1734}
W20260807 08:43:17.165321 7708 catalog_manager.cc:4584] TS 7dc8574703674c17b898ecda0001ab47 (127.28.108.129:42573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3edf21ae46d4ba794ff833ecfbf440f: Network error: Client connection negotiation failed: client connection to 127.28.108.129:42573: connect: Connection refused (error 111)
I20260807 08:43:17.169879 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 8093
I20260807 08:43:17.178023 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 7692
2026-08-07T08:43:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2452 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:43:17 dist-test-slave-9zdj krb5kdc[8287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:43:17 dist-test-slave-9zdj krb5kdc[8287](info): set up 2 sockets
Aug 07 08:43:17 dist-test-slave-9zdj krb5kdc[8287](info): commencing operation
krb5kdc: starting...
W20260807 08:43:18.343640 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:43:18 dist-test-slave-9zdj krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:43:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:43:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.108.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.108.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/rangeradmin_127.28.108.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/rangeradmin_127.28.108.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.108.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.108.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/rangerlookup_127.28.108.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/rangerlookup_127.28.108.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.108.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.108.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/HTTP_127.28.108.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/HTTP_127.28.108.148@KRBTEST.COM.keytab.
I20260807 08:43:18.574283 29106 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskufh9Yd/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/postgres -l logfile start
2026-08-07 08:43:22.153 UTC [8319] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 08:43:22.154 UTC [8319] LOG: listening on IPv4 address "127.28.108.148", port 36899
2026-08-07 08:43:22.162 UTC [8319] LOG: listening on Unix socket "/tmp/.s.PGSQL.36899"
2026-08-07 08:43:22.205 UTC [8322] LOG: database system was shut down at 2026-08-07 08:43:19 UTC
2026-08-07 08:43:22.211 UTC [8319] LOG: database system is ready to accept connections
I20260807 08:43:23.307926 29106 mini_postgres.cc:96] Postgres bound to 36899
2026-08-07 08:43:23.316 UTC [8330] FATAL: database "slave" does not exist
127.28.108.148:36899 - accepting connections
I20260807 08:43:23.318080 29106 mini_ranger.cc:161] Starting Ranger...
I20260807 08:43:23.339005 29106 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 08:43:24.156502 29106 mini_ranger.cc:87] Created ranger Postgres database
I20260807 08:43:24.156620 29106 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-admin
2026-08-07 08:43:24,491 [I] DB FLAVOR :POSTGRES
2026-08-07 08:43:24,491 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 08:43:24,492 [I] Checking connection..
2026-08-07 08:43:24,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 08:43:24,779 [I] Checking connection passed.
2026-08-07 08:43:24,780 [I] --------- Verifying version history table ---------
2026-08-07 08:43:24,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 08:43:25,075 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 08:43:25,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 08:43:25,363 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 08:43:25,363 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 08:43:25,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 08:43:25.655 UTC [8402] WARNING: there is no transaction in progress
2026-08-07 08:43:25,664 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 08:43:25,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 08:43:25,952 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 08:43:25,952 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 08:43:25,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 08:43:26,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 08:43:26,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9zdj.c.gcp-upstream.internal','N') ;"
2026-08-07 08:43:26,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 08:43:27,130 [I] Table x_portal_user does not exist in database ranger
2026-08-07 08:43:27,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 08:43:27,430 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 08:43:27,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 08:43:27,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 08:43:28,029 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 08:43:28,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 08:43:29.381 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.406 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.430 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.452 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.475 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.501 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.598 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.609 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.631 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.646 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.661 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.681 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.695 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.705 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.716 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:29.728 UTC [8516] WARNING: there is no transaction in progress
2026-08-07 08:43:30,897 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 08:43:30,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 08:43:31,197 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 08:43:31,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 08:43:31,500 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 08:43:31,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 08:43:31,809 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 08:43:31,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 08:43:32,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 08:43:32,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9zdj.c.gcp-upstream.internal';"
2026-08-07 08:43:32,681 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 08:43:32,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 08:43:32,975 [I] DB_PATCHES have already been applied
I20260807 08:43:32.981842 29106 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-admin:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskufh9Yd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 08:43:32.981933 29106 mini_ranger.cc:193] Using host: 127.28.108.148
I20260807 08:43:32.986735 29106 mini_ranger.cc:227] Ranger admin URL: http://127.28.108.148:50769
Aug 07, 2026 8:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 8:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 8:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 8:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 8:43:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 8:43:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 8:43:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 08:43:34.303622 29106 mini_ranger.cc:160] Time spent starting Ranger: real 10.986s user 0.002s sys 0.012s
Aug 07, 2026 8:44:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 8:44:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 07, 2026 8:44:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 8:44:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 8:44:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 08:44:03.288151 29106 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.108.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.108.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/rangerkms_127.28.108.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/rangerkms_127.28.108.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.108.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.108.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.108.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/HTTP_127.28.108.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.108.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/HTTP_127.28.108.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 08:44:03 dist-test-slave-9zdj krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092243, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 08:44:03.410946 29106 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 08:44:03.427678 29106 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 08:44:04.666343 29106 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 08:44:04.666460 29106 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-kms
I20260807 08:44:04.666481 29106 mini_ranger_kms.cc:227] Using postgres at 127.28.108.148:36899
2026-08-07 08:44:05,142 [I] DB FLAVOR :POSTGRES
2026-08-07 08:44:05,142 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 08:44:05,142 [I] Checking connection
2026-08-07 08:44:05,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 08:44:05,472 [I] connection success
2026-08-07 08:44:05,472 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 08:44:05,472 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 08:44:05,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 08:44:05,794 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 08:44:05,795 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 08:44:05,795 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 08:44:05,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:36899/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 08:44:06,158 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 08:44:06.824261 29106 mini_ranger_kms.cc:310] Created kms service
I20260807 08:44:07.085820 29106 mini_ranger_kms.cc:326] Created kudu user
I20260807 08:44:07.150243 29106 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 08:44:07 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092243, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:44:07.627224 29106 mini_ranger_kms.cc:384] Added ranger policy
I20260807 08:44:07.627373 29106 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-kms:/tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 08:44:07.627400 29106 mini_ranger_kms.cc:240] Using host: 127.28.108.148
I20260807 08:44:07.632640 29106 mini_ranger_kms.cc:276] Ranger KMS PID: 8712
I20260807 08:44:07.632805 29106 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.108.148:39871
Aug 07, 2026 8:44:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 8:44:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 8:44:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 8:44:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 8:44:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 8:44:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260807 08:44:10.366551 29106 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.956s user 0.000s sys 0.369s
I20260807 08:44:10.366693 29106 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 08:44:10.366815 29106 mini_ranger_kms.cc:401] 127.28.108.148:39871/kms/v1/keys
Aug 07 08:44:10 dist-test-slave-9zdj krb5kdc[8287](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786092250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.108.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 08:44:11 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.108.148@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
Aug 07 08:44:11 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.108.148@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
Aug 07 08:44:14 dist-test-slave-9zdj krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:44:14.644332 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:37437
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:37437
--ranger_config_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:39871/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:44:14.760881 8765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:44:14.761173 8765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:44:14.761202 8765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:44:14.762837 8765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:44:14.763087 8765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:44:14.763437 8765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:44:14.763459 8765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:44:14.763475 8765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:44:14.766182 8765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.108.148:39871/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:37437
--ranger_config_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:37437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:44:14.766542 8765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:44:14.766834 8765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:44:14.774252 8771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:44:14.774231 8765 server_base.cc:1034] running on GCE node
W20260807 08:44:14.774235 8770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:44:14.774387 8773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:44:14.774891 8765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:44:14.775245 8765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:44:14.776449 8765 hybrid_clock.cc:648] HybridClock initialized: now 1786092254776422 us; error 47 us; skew 500 ppm
Aug 07 08:44:14 dist-test-slave-9zdj krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092254, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:44:14.783722 8765 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:44:14.784389 8765 webserver.cc:460] Webserver started at http://127.28.108.190:43069/ using document root <none> and password file <none>
I20260807 08:44:14.784669 8765 fs_manager.cc:359] Metadata directory not provided
I20260807 08:44:14.784745 8765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:44:14.784947 8765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 08:44:14 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092254, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:44:14.847492 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "f2ea142fe559491dad86da9053043d21"
format_stamp: "Formatted at 2026-08-07 08:44:14 on dist-test-slave-9zdj"
server_key: "a62526c3bc6ad1306fcab99dac481c6e"
server_key_iv: "7eb9e5e6dce3db13fcfe63591cefc3f1"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:14.848026 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "f2ea142fe559491dad86da9053043d21"
format_stamp: "Formatted at 2026-08-07 08:44:14 on dist-test-slave-9zdj"
server_key: "a62526c3bc6ad1306fcab99dac481c6e"
server_key_iv: "7eb9e5e6dce3db13fcfe63591cefc3f1"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:14.850370 8765 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 08:44:14.865820 8780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:44:14.866070 8765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:44:14.866199 8765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "f2ea142fe559491dad86da9053043d21"
format_stamp: "Formatted at 2026-08-07 08:44:14 on dist-test-slave-9zdj"
server_key: "a62526c3bc6ad1306fcab99dac481c6e"
server_key_iv: "7eb9e5e6dce3db13fcfe63591cefc3f1"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:14.866336 8765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:44:14.890216 8765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:44:14.892925 8765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:44:14.893150 8765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:44:14.901281 8765 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:37437
I20260807 08:44:14.901309 8836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:37437 every 8 connection(s)
I20260807 08:44:14.902000 8765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:44:14.903239 8837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:44:14.905884 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 8765
I20260807 08:44:14.906032 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:44:14.906229 29106 external_mini_cluster.cc:1428] Setting key 8c0f0ce99640fb1a45e093b786623644
I20260807 08:44:14.906972 8837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21: Bootstrap starting.
I20260807 08:44:14.908034 8837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21: Neither blocks nor log segments found. Creating new log.
I20260807 08:44:14.908444 8837 log.cc:824] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21: Log is configured to *not* fsync() on all Append() calls
I20260807 08:44:14.909981 8837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21: No bootstrap required, opened a new log
I20260807 08:44:14.912784 8837 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ea142fe559491dad86da9053043d21" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 37437 } }
I20260807 08:44:14.913054 8837 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:44:14.913146 8837 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ea142fe559491dad86da9053043d21, State: Initialized, Role: FOLLOWER
I20260807 08:44:14.913395 8837 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ea142fe559491dad86da9053043d21" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 37437 } }
I20260807 08:44:14.913511 8837 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:44:14.913573 8837 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:44:14.913669 8837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:44:14.914665 8837 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ea142fe559491dad86da9053043d21" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 37437 } }
I20260807 08:44:14.914808 8837 leader_election.cc:302] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2ea142fe559491dad86da9053043d21; no voters:
I20260807 08:44:14.915125 8837 leader_election.cc:288] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:44:14.915400 8843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:44:14.915529 8837 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:44:14.915668 8843 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [term 1 LEADER]: Becoming Leader. State: Replica: f2ea142fe559491dad86da9053043d21, State: Running, Role: LEADER
Aug 07 08:44:14 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:44:14.916819 8843 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ea142fe559491dad86da9053043d21" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 37437 } }
I20260807 08:44:14.918254 8844 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2ea142fe559491dad86da9053043d21. Latest consensus state: current_term: 1 leader_uuid: "f2ea142fe559491dad86da9053043d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ea142fe559491dad86da9053043d21" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 37437 } } }
I20260807 08:44:14.918306 8842 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2ea142fe559491dad86da9053043d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2ea142fe559491dad86da9053043d21" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 37437 } } }
I20260807 08:44:14.918385 8844 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [sys.catalog]: This master's current role is: LEADER
I20260807 08:44:14.918403 8842 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21 [sys.catalog]: This master's current role is: LEADER
I20260807 08:44:14.918642 8859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:44:14.918707 8837 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 08:44:14.919466 8859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:44:14.926064 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:14.906941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.28.108.190:37437)
0807 08:44:14.907194 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 08:44:14.907205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:44:14.907233 (+ 28us) server_negotiation.cc:407] Connection header received
0807 08:44:14.907912 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:44:14.907921 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:44:14.907985 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:44:14.908177 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:44:14.909042 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:14.910820 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:14.912096 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:14.912847 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:14.912977 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:14.916456 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:44:14.916461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:44:14.916461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:44:14.916476 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:44:14.919218 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:44:14.919769 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:44:14.919773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:44:14.919774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:44:14.919893 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:44:14.920160 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:44:14.920161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:44:14.920162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:44:14.923029 (+ 2867us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:44:14.924463 (+ 1434us) server_negotiation.cc:1035] Waiting for connection context
0807 08:44:14.924927 (+ 464us) server_negotiation.cc:299] Negotiation successful
0807 08:44:14.925299 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":99,"threads_started":1}
I20260807 08:44:14.926705 8859 catalog_manager.cc:1347] Generated new cluster ID: cef5bae7198d4cb39428490d2d04e449
I20260807 08:44:14.926790 8859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:44:14.940376 8859 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:44:14.941228 8859 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:44:14.946382 8859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21: Generated new TSK 0
I20260807 08:44:14.946664 8859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:44:15.844326 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:39871/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:37437
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:44:15.985809 8886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:44:15.986086 8886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:44:15.986131 8886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:44:15.987596 8886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:44:15.987679 8886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:44:15.987736 8886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:44:15.990151 8886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.108.148:39871/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:37437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:44:15.990478 8886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:44:15.990864 8886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:44:15.999341 8895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:44:15.999426 8893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:44:15.999655 8886 server_base.cc:1034] running on GCE node
W20260807 08:44:15.999472 8892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:44:15.999977 8886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:44:16.000344 8886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:44:16.001529 8886 hybrid_clock.cc:648] HybridClock initialized: now 1786092256001496 us; error 48 us; skew 500 ppm
Aug 07 08:44:15 dist-test-slave-9zdj krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092255, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:44:16.005766 8886 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:44:16.006531 8886 webserver.cc:460] Webserver started at http://127.28.108.129:34271/ using document root <none> and password file <none>
I20260807 08:44:16.006867 8886 fs_manager.cc:359] Metadata directory not provided
I20260807 08:44:16.006960 8886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:44:16.007153 8886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 08:44:16 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092255, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:44:16.023697 8886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "2d483c203d254198bd0677acae43cb79"
format_stamp: "Formatted at 2026-08-07 08:44:16 on dist-test-slave-9zdj"
server_key: "83711b984dde94508e5a5c4ac3765c0e"
server_key_iv: "1333d20d5df9ad09a332073eb2757f23"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:16.024361 8886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "2d483c203d254198bd0677acae43cb79"
format_stamp: "Formatted at 2026-08-07 08:44:16 on dist-test-slave-9zdj"
server_key: "83711b984dde94508e5a5c4ac3765c0e"
server_key_iv: "1333d20d5df9ad09a332073eb2757f23"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:16.028358 8886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260807 08:44:16.053809 8904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:44:16.054121 8886 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260807 08:44:16.054220 8886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "2d483c203d254198bd0677acae43cb79"
format_stamp: "Formatted at 2026-08-07 08:44:16 on dist-test-slave-9zdj"
server_key: "83711b984dde94508e5a5c4ac3765c0e"
server_key_iv: "1333d20d5df9ad09a332073eb2757f23"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:16.054332 8886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:44:16.071750 8886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:44:16.074905 8886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:44:16.075145 8886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:44:16.075440 8886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:44:16.076023 8886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:44:16.076092 8886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:44:16.076143 8886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:44:16.076171 8886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:44:16.087649 8886 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:35467
I20260807 08:44:16.087873 9029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:35467 every 8 connection(s)
I20260807 08:44:16.088199 8886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
Aug 07 08:44:16 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092255, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:44:16.096724 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 8886
I20260807 08:44:16.096863 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:44:16.097123 29106 external_mini_cluster.cc:1428] Setting key a95b31b267f4be7aa4707660e95c7624
I20260807 08:44:16.103102 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.089147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:37437 (local address 127.28.108.129:55193)
0807 08:44:16.089376 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 08:44:16.089428 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 08:44:16.090063 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:44:16.090399 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:44:16.090403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:44:16.090650 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:44:16.091026 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.091034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.092038 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.092039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.093054 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.093057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.093663 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.093664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.093855 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.094613 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:44:16.094627 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:44:16.096976 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:44:16.100103 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:44:16.100120 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:44:16.100124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:44:16.100601 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:44:16.100903 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:44:16.100904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:44:16.100905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:44:16.101096 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:44:16.101479 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:44:16.101487 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:44:16.102200 (+ 713us) client_negotiation.cc:769] Sending connection context
0807 08:44:16.102499 (+ 299us) client_negotiation.cc:240] Negotiation successful
0807 08:44:16.102773 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":125,"threads_started":1}
I20260807 08:44:16.103788 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.089199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:55193 (local address 127.28.108.190:37437)
0807 08:44:16.089367 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 08:44:16.089370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:44:16.089558 (+ 188us) server_negotiation.cc:407] Connection header received
0807 08:44:16.090204 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:44:16.090206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:44:16.090222 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:44:16.090326 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:44:16.091161 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.091908 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.093165 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.093470 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.093548 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.097155 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:44:16.097157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:44:16.097158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:44:16.097170 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:44:16.099952 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:44:16.100714 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:44:16.100715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:44:16.100716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:44:16.100820 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:44:16.101175 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:44:16.101176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:44:16.101177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:44:16.101353 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:44:16.101445 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 08:44:16.103490 (+ 2045us) server_negotiation.cc:299] Negotiation successful
0807 08:44:16.103624 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260807 08:44:16.104326 9030 heartbeater.cc:347] Connected to a master server at 127.28.108.190:37437
I20260807 08:44:16.104482 9030 heartbeater.cc:464] Registering TS with master...
I20260807 08:44:16.104782 9030 heartbeater.cc:511] Master 127.28.108.190:37437 requested a full tablet report, sending...
I20260807 08:44:16.105991 8798 ts_manager.cc:194] Registered new tserver with Master: 2d483c203d254198bd0677acae43cb79 (127.28.108.129:35467)
I20260807 08:44:16.107642 8798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:55193
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:44:16.149904 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:39871/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:37437
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 08:44:16 dist-test-slave-9zdj krb5kdc[8287](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786092256, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 08:44:16.274008 9038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:44:16.274421 9038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:44:16.274470 9038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:44:16.276772 9038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:44:16.276894 9038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:44:16.276942 9038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:44:16.280047 9038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.108.148:39871/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:37437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:44:16.280467 9038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:44:16.281059 9038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:44:16.290365 9046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:44:16.290691 9044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:44:16.291175 9043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:44:16.292140 9038 server_base.cc:1034] running on GCE node
I20260807 08:44:16.292559 9038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:44:16.292943 9038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:44:16.294106 9038 hybrid_clock.cc:648] HybridClock initialized: now 1786092256294082 us; error 44 us; skew 500 ppm
Aug 07 08:44:16 dist-test-slave-9zdj krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092256, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:44:16.297719 9038 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:44:16.298332 9038 webserver.cc:460] Webserver started at http://127.28.108.130:43529/ using document root <none> and password file <none>
I20260807 08:44:16.298717 9038 fs_manager.cc:359] Metadata directory not provided
I20260807 08:44:16.298779 9038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:44:16.298919 9038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 08:44:16 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092256, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:44:16.316903 9038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "0d8fc35160ef41459458e03f6e94003a"
format_stamp: "Formatted at 2026-08-07 08:44:16 on dist-test-slave-9zdj"
server_key: "a918afaafb42b22c668a2265d582d8d2"
server_key_iv: "72ea4e7a145431a10054e52061700f81"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:16.317376 9038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "0d8fc35160ef41459458e03f6e94003a"
format_stamp: "Formatted at 2026-08-07 08:44:16 on dist-test-slave-9zdj"
server_key: "a918afaafb42b22c668a2265d582d8d2"
server_key_iv: "72ea4e7a145431a10054e52061700f81"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:16.325906 9038 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.001s
I20260807 08:44:16.344406 9054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:44:16.344659 9038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:44:16.344775 9038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "0d8fc35160ef41459458e03f6e94003a"
format_stamp: "Formatted at 2026-08-07 08:44:16 on dist-test-slave-9zdj"
server_key: "a918afaafb42b22c668a2265d582d8d2"
server_key_iv: "72ea4e7a145431a10054e52061700f81"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:16.344915 9038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:44:16.356562 9038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:44:16.360145 9038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:44:16.360348 9038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:44:16.360630 9038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:44:16.361057 9038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:44:16.361111 9038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:44:16.361145 9038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:44:16.361164 9038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:44:16.370711 9038 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:44027
I20260807 08:44:16.371243 9038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:44:16.371719 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 9038
I20260807 08:44:16.371845 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:44:16.372118 29106 external_mini_cluster.cc:1428] Setting key 83328580d16898064ca0084fffa8f2f8
I20260807 08:44:16.377914 9175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:44027 every 8 connection(s)
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Aug 07 08:44:16 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092256, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:44:16.399255 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.383255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:39469 (local address 127.28.108.190:37437)
0807 08:44:16.383317 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0807 08:44:16.383320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:44:16.383710 (+ 390us) server_negotiation.cc:407] Connection header received
0807 08:44:16.384648 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:44:16.384650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:44:16.384675 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:44:16.384715 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:44:16.386127 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.387015 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.389035 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.389427 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.389485 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.393119 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:44:16.393122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:44:16.393122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:44:16.393135 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:44:16.395599 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:44:16.396419 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:44:16.396421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:44:16.396422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:44:16.396505 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:44:16.396912 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:44:16.396913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:44:16.396914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:44:16.397127 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:44:16.397228 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 08:44:16.398938 (+ 1710us) server_negotiation.cc:299] Negotiation successful
0807 08:44:16.399081 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 08:44:16.399436 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.382786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:37437 (local address 127.28.108.130:39469)
0807 08:44:16.383523 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0807 08:44:16.383588 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 08:44:16.384478 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:44:16.384981 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:44:16.384986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:44:16.385423 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:44:16.385961 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.385972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.387182 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.387183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.388869 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.388874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.389606 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.389607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.389824 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.390603 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:44:16.390616 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:44:16.392935 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:44:16.395750 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:44:16.395762 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:44:16.395766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:44:16.396304 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:44:16.396609 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:44:16.396610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:44:16.396610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:44:16.396820 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:44:16.397224 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:44:16.397231 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:44:16.397453 (+ 222us) client_negotiation.cc:769] Sending connection context
0807 08:44:16.398882 (+ 1429us) client_negotiation.cc:240] Negotiation successful
0807 08:44:16.399123 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":142,"threads_started":1}
I20260807 08:44:16.400115 9176 heartbeater.cc:347] Connected to a master server at 127.28.108.190:37437
I20260807 08:44:16.400265 9176 heartbeater.cc:464] Registering TS with master...
I20260807 08:44:16.400542 9176 heartbeater.cc:511] Master 127.28.108.190:37437 requested a full tablet report, sending...
I20260807 08:44:16.401064 8798 ts_manager.cc:194] Registered new tserver with Master: 0d8fc35160ef41459458e03f6e94003a (127.28.108.130:44027)
I20260807 08:44:16.401758 8798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:39469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:44:16.414846 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:39871/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:37437
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:44:16.563997 9181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:44:16.564404 9181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:44:16.564452 9181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:44:16.567116 9181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:44:16.567233 9181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:44:16.567291 9181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:44:16.570644 9181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.108.148:39871/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:37437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:44:16.571221 9181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:44:16.571573 9181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:44:16.580960 9191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:44:16.581108 9189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:44:16.581342 9188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:44:16.582049 9181 server_base.cc:1034] running on GCE node
I20260807 08:44:16.582410 9181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:44:16.582808 9181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:44:16.584025 9181 hybrid_clock.cc:648] HybridClock initialized: now 1786092256583997 us; error 47 us; skew 500 ppm
Aug 07 08:44:16 dist-test-slave-9zdj krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092256, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:44:16.588469 9181 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:44:16.589146 9181 webserver.cc:460] Webserver started at http://127.28.108.131:39911/ using document root <none> and password file <none>
I20260807 08:44:16.589856 9181 fs_manager.cc:359] Metadata directory not provided
I20260807 08:44:16.589926 9181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:44:16.590059 9181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 08:44:16 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092256, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:44:16.607827 9181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "71ca1649a1c240c4b772bf1b46b56f6a"
format_stamp: "Formatted at 2026-08-07 08:44:16 on dist-test-slave-9zdj"
server_key: "be76b35e320b9d1a75ad8502ab6d1cd9"
server_key_iv: "17a9155279da0179bcf67650e406908c"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:16.608450 9181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "71ca1649a1c240c4b772bf1b46b56f6a"
format_stamp: "Formatted at 2026-08-07 08:44:16 on dist-test-slave-9zdj"
server_key: "be76b35e320b9d1a75ad8502ab6d1cd9"
server_key_iv: "17a9155279da0179bcf67650e406908c"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:16.610963 9181 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.004s
I20260807 08:44:16.629822 9199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:44:16.630154 9181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 08:44:16.630282 9181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "71ca1649a1c240c4b772bf1b46b56f6a"
format_stamp: "Formatted at 2026-08-07 08:44:16 on dist-test-slave-9zdj"
server_key: "be76b35e320b9d1a75ad8502ab6d1cd9"
server_key_iv: "17a9155279da0179bcf67650e406908c"
server_key_version: "kuduclusterkey@0"
I20260807 08:44:16.630398 9181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:44:16.668581 9181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:44:16.671576 9181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:44:16.671885 9181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:44:16.672204 9181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:44:16.672706 9181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:44:16.672807 9181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:44:16.672886 9181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:44:16.673041 9181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:44:16.683174 9181 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:37845
I20260807 08:44:16.683981 9181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:44:16.686383 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 9181
I20260807 08:44:16.686524 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:44:16.686903 29106 external_mini_cluster.cc:1428] Setting key 945c99741821b7305f87af28814736f3
I20260807 08:44:16.689608 9321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:37845 every 8 connection(s)
Aug 07 08:44:16 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092256, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:44:16.700490 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.690337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:57559 (local address 127.28.108.190:37437)
0807 08:44:16.690409 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 08:44:16.690412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:44:16.690831 (+ 419us) server_negotiation.cc:407] Connection header received
0807 08:44:16.691523 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:44:16.691524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:44:16.691539 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:44:16.691581 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:44:16.692292 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.692812 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.694001 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.694207 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.694359 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.696700 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:44:16.696702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:44:16.696704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:44:16.696717 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:44:16.698598 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:44:16.699348 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:44:16.699349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:44:16.699350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:44:16.699454 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:44:16.699712 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:44:16.699713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:44:16.699713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:44:16.699850 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:44:16.699935 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 08:44:16.700294 (+ 359us) server_negotiation.cc:299] Negotiation successful
0807 08:44:16.700391 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 08:44:16.700721 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.690344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:37437 (local address 127.28.108.131:57559)
0807 08:44:16.690625 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0807 08:44:16.690672 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 08:44:16.691398 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:44:16.691675 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:44:16.691680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:44:16.691953 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:44:16.692172 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.692176 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.692936 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.692938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.693905 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.693908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.694368 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.694370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.694506 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.695109 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:44:16.695119 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:44:16.696581 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:44:16.698707 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:44:16.698717 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:44:16.698720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:44:16.699257 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:44:16.699519 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:44:16.699520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:44:16.699520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:44:16.699638 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:44:16.699925 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:44:16.699932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:44:16.700103 (+ 171us) client_negotiation.cc:769] Sending connection context
0807 08:44:16.700243 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 08:44:16.700429 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":199,"threads_started":1}
I20260807 08:44:16.701314 9323 heartbeater.cc:347] Connected to a master server at 127.28.108.190:37437
I20260807 08:44:16.701447 9323 heartbeater.cc:464] Registering TS with master...
I20260807 08:44:16.701716 9323 heartbeater.cc:511] Master 127.28.108.190:37437 requested a full tablet report, sending...
I20260807 08:44:16.702136 8798 ts_manager.cc:194] Registered new tserver with Master: 71ca1649a1c240c4b772bf1b46b56f6a (127.28.108.131:37845)
I20260807 08:44:16.702641 8798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:57559
I20260807 08:44:16.710659 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:44:16.725327 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.714183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.28.108.190:37437)
0807 08:44:16.714243 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 08:44:16.714246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:44:16.714805 (+ 559us) server_negotiation.cc:407] Connection header received
0807 08:44:16.714940 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:44:16.714942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:44:16.714958 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:44:16.715048 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:44:16.716388 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.717314 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.718697 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.719036 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.719140 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.721115 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:44:16.721116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:44:16.721117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:44:16.721128 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:44:16.723416 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:44:16.724011 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:44:16.724013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:44:16.724013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:44:16.724106 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:44:16.724423 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:44:16.724424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:44:16.724424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:44:16.724626 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:44:16.724748 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 08:44:16.724951 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 08:44:16.725062 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 08:44:16.727584 8798 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:44:16.730441 8798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:44:16.741070 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.734430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.28.108.131:37845)
0807 08:44:16.734636 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 08:44:16.734642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:44:16.735282 (+ 640us) server_negotiation.cc:407] Connection header received
0807 08:44:16.735614 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:44:16.735630 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:44:16.735650 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:44:16.735717 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:44:16.736060 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.737394 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.739498 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.740429 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.740511 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.740585 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 08:44:16.740910 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 08:44:16.740962 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
I20260807 08:44:16.741793 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.734105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:35467 (local address 127.0.0.1:37436)
0807 08:44:16.736081 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0807 08:44:16.736092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:44:16.736140 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:44:16.736432 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:44:16.736432 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:44:16.736434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:44:16.736478 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.736480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.738360 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.738362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.740244 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.740247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.741449 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.741451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.741535 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.741539 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:44:16.741595 (+ 56us) client_negotiation.cc:240] Negotiation successful
0807 08:44:16.741685 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1958,"mutex_wait_us":2,"thread_start_us":121,"threads_started":1}
I20260807 08:44:16.742205 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.734104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:37845 (local address 127.0.0.1:52952)
0807 08:44:16.735149 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0807 08:44:16.735200 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 08:44:16.735464 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:44:16.735874 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:44:16.735877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:44:16.735880 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:44:16.735960 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.735962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.737512 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.737513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.739225 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.739229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.740659 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.740660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.740758 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.740777 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 08:44:16.741551 (+ 774us) client_negotiation.cc:240] Negotiation successful
0807 08:44:16.741704 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":92,"threads_started":1}
I20260807 08:44:16.742802 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.734837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.28.108.129:35467)
0807 08:44:16.735412 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0807 08:44:16.735417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:44:16.736203 (+ 786us) server_negotiation.cc:407] Connection header received
0807 08:44:16.736248 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:44:16.736250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:44:16.736269 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:44:16.736352 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:44:16.736570 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.738213 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.740363 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.741249 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.741337 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.741404 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 08:44:16.742126 (+ 722us) server_negotiation.cc:299] Negotiation successful
0807 08:44:16.742189 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":73,"threads_started":1}
I20260807 08:44:16.743350 9240 tablet_service.cc:1467] Processing CreateTablet for tablet 91c252dec6ed4d90b94e5c788418b3ab (DEFAULT_TABLE table=test-table [id=c0ba3136cead47fc850b37a904017dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:44:16.743542 8959 tablet_service.cc:1467] Processing CreateTablet for tablet 91c252dec6ed4d90b94e5c788418b3ab (DEFAULT_TABLE table=test-table [id=c0ba3136cead47fc850b37a904017dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:44:16.743775 9240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c252dec6ed4d90b94e5c788418b3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:44:16.743956 8959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c252dec6ed4d90b94e5c788418b3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:44:16.749517 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.734404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:44027 (local address 127.0.0.1:34226)
0807 08:44:16.741712 (+ 7308us) negotiation.cc:107] Waiting for socket to connect
0807 08:44:16.741727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:44:16.741803 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:44:16.742280 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:44:16.742282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:44:16.742285 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:44:16.742355 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.742357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.743876 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.743877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.747636 (+ 3759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.747643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.749195 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.749196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.749275 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.749282 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 08:44:16.749348 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 08:44:16.749414 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7285,"thread_start_us":80,"threads_started":1}
I20260807 08:44:16.749852 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.734782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.28.108.130:44027)
0807 08:44:16.734996 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 08:44:16.735002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:44:16.741896 (+ 6894us) server_negotiation.cc:407] Connection header received
0807 08:44:16.742033 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:44:16.742037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:44:16.742060 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:44:16.742145 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:44:16.742451 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.743736 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.747848 (+ 4112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.748931 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.749062 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.749140 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 08:44:16.749697 (+ 557us) server_negotiation.cc:299] Negotiation successful
0807 08:44:16.749749 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
I20260807 08:44:16.749575 9342 tablet_bootstrap.cc:492] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79: Bootstrap starting.
I20260807 08:44:16.750870 9106 tablet_service.cc:1467] Processing CreateTablet for tablet 91c252dec6ed4d90b94e5c788418b3ab (DEFAULT_TABLE table=test-table [id=c0ba3136cead47fc850b37a904017dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:44:16.751394 9106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c252dec6ed4d90b94e5c788418b3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:44:16.751580 9342 tablet_bootstrap.cc:654] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79: Neither blocks nor log segments found. Creating new log.
I20260807 08:44:16.752014 9342 log.cc:824] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79: Log is configured to *not* fsync() on all Append() calls
I20260807 08:44:16.752688 9343 tablet_bootstrap.cc:492] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a: Bootstrap starting.
I20260807 08:44:16.754550 9343 tablet_bootstrap.cc:654] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a: Neither blocks nor log segments found. Creating new log.
I20260807 08:44:16.755123 9343 log.cc:824] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a: Log is configured to *not* fsync() on all Append() calls
I20260807 08:44:16.756304 9342 tablet_bootstrap.cc:492] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79: No bootstrap required, opened a new log
I20260807 08:44:16.756492 9342 ts_tablet_manager.cc:1397] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260807 08:44:16.760617 9342 raft_consensus.cc:348] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37845 } } peers { permanent_uuid: "2d483c203d254198bd0677acae43cb79" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35467 } } peers { permanent_uuid: "0d8fc35160ef41459458e03f6e94003a" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 44027 } }
I20260807 08:44:16.761759 9343 tablet_bootstrap.cc:492] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a: No bootstrap required, opened a new log
I20260807 08:44:16.761947 9343 ts_tablet_manager.cc:1397] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260807 08:44:16.765007 9343 raft_consensus.cc:348] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37845 } } peers { permanent_uuid: "2d483c203d254198bd0677acae43cb79" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35467 } } peers { permanent_uuid: "0d8fc35160ef41459458e03f6e94003a" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 44027 } }
I20260807 08:44:16.765429 9342 raft_consensus.cc:374] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:44:16.765569 9342 raft_consensus.cc:729] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d483c203d254198bd0677acae43cb79, State: Initialized, Role: FOLLOWER
I20260807 08:44:16.765902 9342 consensus_queue.cc:260] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37845 } } peers { permanent_uuid: "2d483c203d254198bd0677acae43cb79" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35467 } } peers { permanent_uuid: "0d8fc35160ef41459458e03f6e94003a" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 44027 } }
I20260807 08:44:16.766587 9342 ts_tablet_manager.cc:1428] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260807 08:44:16.766806 9030 heartbeater.cc:503] Master 127.28.108.190:37437 was elected leader, sending a full tablet report...
I20260807 08:44:16.768085 9343 raft_consensus.cc:374] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:44:16.768297 9343 raft_consensus.cc:729] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ca1649a1c240c4b772bf1b46b56f6a, State: Initialized, Role: FOLLOWER
I20260807 08:44:16.768858 9343 consensus_queue.cc:260] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37845 } } peers { permanent_uuid: "2d483c203d254198bd0677acae43cb79" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35467 } } peers { permanent_uuid: "0d8fc35160ef41459458e03f6e94003a" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 44027 } }
I20260807 08:44:16.769443 9343 ts_tablet_manager.cc:1428] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260807 08:44:16.769713 9323 heartbeater.cc:503] Master 127.28.108.190:37437 was elected leader, sending a full tablet report...
I20260807 08:44:16.769904 9346 tablet_bootstrap.cc:492] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a: Bootstrap starting.
I20260807 08:44:16.771428 9346 tablet_bootstrap.cc:654] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a: Neither blocks nor log segments found. Creating new log.
I20260807 08:44:16.771862 9346 log.cc:824] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a: Log is configured to *not* fsync() on all Append() calls
I20260807 08:44:16.773223 9346 tablet_bootstrap.cc:492] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a: No bootstrap required, opened a new log
I20260807 08:44:16.773343 9346 ts_tablet_manager.cc:1397] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 08:44:16.776907 9346 raft_consensus.cc:348] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37845 } } peers { permanent_uuid: "2d483c203d254198bd0677acae43cb79" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35467 } } peers { permanent_uuid: "0d8fc35160ef41459458e03f6e94003a" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 44027 } }
I20260807 08:44:16.777163 9346 raft_consensus.cc:374] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:44:16.777207 9346 raft_consensus.cc:729] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8fc35160ef41459458e03f6e94003a, State: Initialized, Role: FOLLOWER
I20260807 08:44:16.777393 9346 consensus_queue.cc:260] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37845 } } peers { permanent_uuid: "2d483c203d254198bd0677acae43cb79" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35467 } } peers { permanent_uuid: "0d8fc35160ef41459458e03f6e94003a" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 44027 } }
I20260807 08:44:16.777801 9346 ts_tablet_manager.cc:1428] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260807 08:44:16.777858 9176 heartbeater.cc:503] Master 127.28.108.190:37437 was elected leader, sending a full tablet report...
W20260807 08:44:16.839846 9031 tablet.cc:2369] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 08:44:16 dist-test-slave-9zdj krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092256, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
W20260807 08:44:16.883980 9177 tablet.cc:2369] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:44:16.901455 9348 raft_consensus.cc:482] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:44:16.901656 9348 raft_consensus.cc:504] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37845 } } peers { permanent_uuid: "2d483c203d254198bd0677acae43cb79" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35467 } } peers { permanent_uuid: "0d8fc35160ef41459458e03f6e94003a" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 44027 } }
I20260807 08:44:16.902433 9348 leader_election.cc:288] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d483c203d254198bd0677acae43cb79 (127.28.108.129:35467), 0d8fc35160ef41459458e03f6e94003a (127.28.108.130:44027)
I20260807 08:44:16.907433 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.902709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:35467 (local address 127.28.108.131:34943)
0807 08:44:16.902859 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 08:44:16.902871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:44:16.902919 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:44:16.903639 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:44:16.903640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:44:16.903642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:44:16.903822 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.903825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.904776 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.904777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.906016 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.906019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.907054 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.907055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.907139 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.907143 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:44:16.907188 (+ 45us) client_negotiation.cc:240] Negotiation successful
0807 08:44:16.907317 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260807 08:44:16.907945 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.903032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:34943 (local address 127.28.108.129:35467)
0807 08:44:16.903074 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0807 08:44:16.903076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:44:16.903083 (+ 7us) server_negotiation.cc:407] Connection header received
0807 08:44:16.903112 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:44:16.903113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:44:16.903123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:44:16.903177 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:44:16.903934 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.904671 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.906259 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.906918 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.906983 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.906994 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:44:16.907773 (+ 779us) server_negotiation.cc:299] Negotiation successful
0807 08:44:16.907839 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:44:16.908105 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.903553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:55439 (local address 127.28.108.130:44027)
0807 08:44:16.903601 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 08:44:16.903603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:44:16.904315 (+ 712us) server_negotiation.cc:407] Connection header received
0807 08:44:16.904415 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:44:16.904416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:44:16.904426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:44:16.904455 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:44:16.904662 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.905411 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.906809 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.907528 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.907612 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.907623 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:44:16.907945 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 08:44:16.908021 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 08:44:16.908310 8980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c252dec6ed4d90b94e5c788418b3ab" candidate_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d483c203d254198bd0677acae43cb79" is_pre_election: true
I20260807 08:44:16.908535 8980 raft_consensus.cc:2454] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ca1649a1c240c4b772bf1b46b56f6a in term 0.
I20260807 08:44:16.908546 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.903330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:44027 (local address 127.28.108.131:55439)
0807 08:44:16.904240 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0807 08:44:16.904252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:44:16.904342 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:44:16.904520 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:44:16.904520 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:44:16.904522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:44:16.904597 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.904598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.905516 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.905516 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.906697 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:44:16.906700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.907742 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.907743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:44:16.907839 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.907844 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:44:16.908365 (+ 521us) client_negotiation.cc:240] Negotiation successful
0807 08:44:16.908435 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":116,"threads_started":1}
I20260807 08:44:16.908950 9200 leader_election.cc:302] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d483c203d254198bd0677acae43cb79, 71ca1649a1c240c4b772bf1b46b56f6a; no voters:
I20260807 08:44:16.908946 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c252dec6ed4d90b94e5c788418b3ab" candidate_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d8fc35160ef41459458e03f6e94003a" is_pre_election: true
I20260807 08:44:16.909122 9114 raft_consensus.cc:2454] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ca1649a1c240c4b772bf1b46b56f6a in term 0.
I20260807 08:44:16.909155 9348 raft_consensus.cc:2781] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:44:16.909293 9348 raft_consensus.cc:482] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:44:16.909318 9348 raft_consensus.cc:3037] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:44:16.910458 9348 raft_consensus.cc:504] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37845 } } peers { permanent_uuid: "2d483c203d254198bd0677acae43cb79" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35467 } } peers { permanent_uuid: "0d8fc35160ef41459458e03f6e94003a" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 44027 } }
I20260807 08:44:16.910647 9348 leader_election.cc:288] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [CANDIDATE]: Term 1 election: Requested vote from peers 2d483c203d254198bd0677acae43cb79 (127.28.108.129:35467), 0d8fc35160ef41459458e03f6e94003a (127.28.108.130:44027)
I20260807 08:44:16.910902 8980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c252dec6ed4d90b94e5c788418b3ab" candidate_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d483c203d254198bd0677acae43cb79"
I20260807 08:44:16.910902 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c252dec6ed4d90b94e5c788418b3ab" candidate_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d8fc35160ef41459458e03f6e94003a"
I20260807 08:44:16.910995 9114 raft_consensus.cc:3037] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:44:16.910995 8980 raft_consensus.cc:3037] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:44:16.912163 9114 raft_consensus.cc:2454] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ca1649a1c240c4b772bf1b46b56f6a in term 1.
I20260807 08:44:16.912222 8980 raft_consensus.cc:2454] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ca1649a1c240c4b772bf1b46b56f6a in term 1.
I20260807 08:44:16.912470 9200 leader_election.cc:302] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d483c203d254198bd0677acae43cb79, 71ca1649a1c240c4b772bf1b46b56f6a; no voters:
I20260807 08:44:16.912655 9348 raft_consensus.cc:2781] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:44:16.913012 9348 raft_consensus.cc:686] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 1 LEADER]: Becoming Leader. State: Replica: 71ca1649a1c240c4b772bf1b46b56f6a, State: Running, Role: LEADER
I20260807 08:44:16.913139 9348 consensus_queue.cc:237] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37845 } } peers { permanent_uuid: "2d483c203d254198bd0677acae43cb79" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35467 } } peers { permanent_uuid: "0d8fc35160ef41459458e03f6e94003a" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 44027 } }
I20260807 08:44:16.914167 8797 catalog_manager.cc:5495] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 71ca1649a1c240c4b772bf1b46b56f6a (127.28.108.131). New cstate: current_term: 1 leader_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71ca1649a1c240c4b772bf1b46b56f6a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 37845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d483c203d254198bd0677acae43cb79" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d8fc35160ef41459458e03f6e94003a" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 44027 } health_report { overall_health: UNKNOWN } } }
I20260807 08:44:16.929226 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.925317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.28.108.131:37845)
0807 08:44:16.925395 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 08:44:16.925397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:44:16.925408 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:44:16.925480 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:44:16.925481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:44:16.925508 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:44:16.925661 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:44:16.925888 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.926808 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.928009 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:44:16.928466 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:44:16.928560 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:44:16.928783 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:44:16.928954 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:44:16.929072 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 08:44:16.929095 (+ 23us) server_negotiation.cc:299] Negotiation successful
0807 08:44:16.929126 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 08:44:16.931897 9114 raft_consensus.cc:1261] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a [term 1 FOLLOWER]: Refusing update from remote peer 71ca1649a1c240c4b772bf1b46b56f6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:44:16.931941 8980 raft_consensus.cc:1261] T 91c252dec6ed4d90b94e5c788418b3ab P 2d483c203d254198bd0677acae43cb79 [term 1 FOLLOWER]: Refusing update from remote peer 71ca1649a1c240c4b772bf1b46b56f6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:44:16.932453 9353 consensus_queue.cc:1035] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d8fc35160ef41459458e03f6e94003a" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 44027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:44:16.932734 9348 consensus_queue.cc:1035] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d483c203d254198bd0677acae43cb79" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:44:16.935835 9356 mvcc.cc:205] Tried to move back new op lower bound from 7315833884389380096 to 7315833884317106176. Current Snapshot: MvccSnapshot[applied={T|T < 7315833884389380096}]
I20260807 08:44:16.938136 9357 mvcc.cc:205] Tried to move back new op lower bound from 7315833884389380096 to 7315833884317106176. Current Snapshot: MvccSnapshot[applied={T|T < 7315833884389380096}]
I20260807 08:44:16.941399 8797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:44:16.941519 8797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:44:16.942579 8797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2ea142fe559491dad86da9053043d21: Sending DeleteTablet for 3 replicas of tablet 91c252dec6ed4d90b94e5c788418b3ab
I20260807 08:44:16.942999 8959 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c252dec6ed4d90b94e5c788418b3ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:44:16 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:37436
I20260807 08:44:16.943063 9106 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c252dec6ed4d90b94e5c788418b3ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:44:16 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:34226
I20260807 08:44:16.943730 9240 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c252dec6ed4d90b94e5c788418b3ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:44:16 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:52952
I20260807 08:44:16.944373 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 8886
I20260807 08:44:16.950045 9369 tablet_replica.cc:330] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a: stopping tablet replica
I20260807 08:44:16.950289 9369 raft_consensus.cc:2229] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:44:16.950562 9369 raft_consensus.cc:2258] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:44:16.951324 9369 ts_tablet_manager.cc:1905] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:44:16.951560 9368 tablet_replica.cc:330] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a: stopping tablet replica
I20260807 08:44:16.951745 9368 raft_consensus.cc:2229] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:44:16.954262 9369 ts_tablet_manager.cc:1918] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:44:16.954587 9368 raft_consensus.cc:2258] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:44:16.954943 9369 log.cc:1196] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/91c252dec6ed4d90b94e5c788418b3ab
I20260807 08:44:16.955335 9368 ts_tablet_manager.cc:1905] T 91c252dec6ed4d90b94e5c788418b3ab P 0d8fc35160ef41459458e03f6e94003a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 08:44:16.957384 8781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:35467 (error 108)
W20260807 08:44:16.957131 9200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:35467 (error 108)
I20260807 08:44:16.958297 9369 ts_tablet_manager.cc:1939] T 91c252dec6ed4d90b94e5c788418b3ab P 71ca1649a1c240c4b772bf1b46b56f6a: Deleting consensus metadata
I20260807 08:44:16.958369 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 9038
I20260807 08:44:16.961109 8784 catalog_manager.cc:4841] TS 71ca1649a1c240c4b772bf1b46b56f6a (127.28.108.131:37845): tablet 91c252dec6ed4d90b94e5c788418b3ab (table test-table [id=c0ba3136cead47fc850b37a904017dbf]) successfully deleted
I20260807 08:44:16.965818 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 9181
I20260807 08:44:16.969986 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.969823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:44027 (local address 127.0.0.1:34236)
0807 08:44:16.969873 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0807 08:44:16.969932 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.130:44027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260807 08:44:16.970224 8785 catalog_manager.cc:4584] TS 0d8fc35160ef41459458e03f6e94003a (127.28.108.130:44027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91c252dec6ed4d90b94e5c788418b3ab: Network error: Client connection negotiation failed: client connection to 127.28.108.130:44027: connect: Connection refused (error 111)
I20260807 08:44:16.970438 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:44:16.969829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:35467 (local address 127.0.0.1:37450)
0807 08:44:16.970371 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0807 08:44:16.970410 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":265}
I20260807 08:44:16.975435 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 8765
I20260807 08:44:16.985195 29106 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 08:44:17.331607 29106 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 08:44:17.331737 29106 mini_ranger.cc:66] Stopping Ranger...
I20260807 08:44:17.688143 29106 mini_ranger.cc:68] Stopped Ranger
2026-08-07 08:44:17.688 UTC [8319] LOG: received smart shutdown request
2026-08-07 08:44:17.694 UTC [8319] LOG: background worker "logical replication launcher" (PID 8328) exited with exit code 1
2026-08-07 08:44:17.694 UTC [8323] LOG: shutting down
2026-08-07 08:44:17.935 UTC [8319] LOG: database system is shut down
2026-08-07T08:44:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (60884 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:44:18 dist-test-slave-9zdj krb5kdc[9380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:44:18 dist-test-slave-9zdj krb5kdc[9380](info): set up 2 sockets
Aug 07 08:44:18 dist-test-slave-9zdj krb5kdc[9380](info): commencing operation
krb5kdc: starting...
W20260807 08:44:19.273276 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:44:19 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:44:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.108.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.108.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/rangeradmin_127.28.108.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/rangeradmin_127.28.108.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.108.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.108.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/rangerlookup_127.28.108.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/rangerlookup_127.28.108.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.108.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.108.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/HTTP_127.28.108.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/HTTP_127.28.108.148@KRBTEST.COM.keytab.
I20260807 08:44:19.502641 29106 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskufh9Yd/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/postgres -l logfile start
2026-08-07 08:44:23.021 UTC [9413] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 08:44:23.021 UTC [9413] LOG: listening on IPv4 address "127.28.108.148", port 59975
2026-08-07 08:44:23.028 UTC [9413] LOG: listening on Unix socket "/tmp/.s.PGSQL.59975"
2026-08-07 08:44:23.064 UTC [9416] LOG: database system was shut down at 2026-08-07 08:44:20 UTC
2026-08-07 08:44:23.069 UTC [9413] LOG: database system is ready to accept connections
I20260807 08:44:24.160300 29106 mini_postgres.cc:96] Postgres bound to 59975
2026-08-07 08:44:24.168 UTC [9424] FATAL: database "slave" does not exist
127.28.108.148:59975 - accepting connections
I20260807 08:44:24.168890 29106 mini_ranger.cc:161] Starting Ranger...
I20260807 08:44:24.187899 29106 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 08:44:24.891903 29106 mini_ranger.cc:87] Created ranger Postgres database
I20260807 08:44:24.891986 29106 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-admin
2026-08-07 08:44:25,218 [I] DB FLAVOR :POSTGRES
2026-08-07 08:44:25,218 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 08:44:25,218 [I] Checking connection..
2026-08-07 08:44:25,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 08:44:25,526 [I] Checking connection passed.
2026-08-07 08:44:25,526 [I] --------- Verifying version history table ---------
2026-08-07 08:44:25,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 08:44:25,836 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 08:44:25,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 08:44:26,117 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 08:44:26,118 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 08:44:26,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 08:44:26.418 UTC [9495] WARNING: there is no transaction in progress
2026-08-07 08:44:26,428 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 08:44:26,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 08:44:26,704 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 08:44:26,704 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 08:44:26,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 08:44:26,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 08:44:27,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9zdj.c.gcp-upstream.internal','N') ;"
2026-08-07 08:44:27,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 08:44:27,843 [I] Table x_portal_user does not exist in database ranger
2026-08-07 08:44:27,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 08:44:28,132 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 08:44:28,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 08:44:28,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 08:44:28,722 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 08:44:28,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 08:44:30.021 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.048 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.073 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.096 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.127 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.154 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.233 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.243 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.267 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.281 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.295 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.313 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.327 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.337 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.348 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:30.359 UTC [9617] WARNING: there is no transaction in progress
2026-08-07 08:44:31,406 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 08:44:31,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 08:44:31,696 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 08:44:31,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 08:44:31,997 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 08:44:31,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 08:44:32,319 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 08:44:32,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 08:44:32,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 08:44:32,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9zdj.c.gcp-upstream.internal';"
2026-08-07 08:44:33,196 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 08:44:33,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 08:44:33,474 [I] DB_PATCHES have already been applied
I20260807 08:44:33.480551 29106 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-admin:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskufh9Yd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 08:44:33.480660 29106 mini_ranger.cc:193] Using host: 127.28.108.148
I20260807 08:44:33.485528 29106 mini_ranger.cc:227] Ranger admin URL: http://127.28.108.148:40393
Aug 07, 2026 8:44:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 8:44:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 8:44:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 8:44:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 8:44:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 8:44:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 8:44:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 08:44:34.788308 29106 mini_ranger.cc:160] Time spent starting Ranger: real 10.619s user 0.000s sys 0.013s
Aug 07, 2026 8:45:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 8:45:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 8:45:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 8:45:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 8:45:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 08:45:02.564080 29106 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.108.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.108.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/rangerkms_127.28.108.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.108.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/rangerkms_127.28.108.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.108.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.108.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.108.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/HTTP_127.28.108.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.108.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/HTTP_127.28.108.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 08:45:02 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092302, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 08:45:02.695490 29106 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 08:45:02.710741 29106 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 08:45:04.027230 29106 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 08:45:04.027314 29106 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-kms
I20260807 08:45:04.027328 29106 mini_ranger_kms.cc:227] Using postgres at 127.28.108.148:59975
2026-08-07 08:45:04,217 [I] DB FLAVOR :POSTGRES
2026-08-07 08:45:04,218 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 08:45:04,218 [I] Checking connection
2026-08-07 08:45:04,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 08:45:04,497 [I] connection success
2026-08-07 08:45:04,498 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 08:45:04,498 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 08:45:04,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 08:45:04,765 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 08:45:04,765 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 08:45:04,766 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 08:45:04,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.108.148:59975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 08:45:05,078 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 08:45:05.689961 29106 mini_ranger_kms.cc:310] Created kms service
I20260807 08:45:05.922319 29106 mini_ranger_kms.cc:326] Created kudu user
I20260807 08:45:05.979780 29106 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 08:45:05 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092302, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:45:06.453752 29106 mini_ranger_kms.cc:384] Added ranger policy
I20260807 08:45:06.454056 29106 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-kms:/tmp/dist-test-taskufh9Yd/build/release/bin/postgresql.jar:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskufh9Yd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 08:45:06.454084 29106 mini_ranger_kms.cc:240] Using host: 127.28.108.148
I20260807 08:45:06.458791 29106 mini_ranger_kms.cc:276] Ranger KMS PID: 9816
I20260807 08:45:06.458909 29106 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.108.148:44011
Aug 07, 2026 8:45:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 8:45:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 8:45:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 8:45:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 8:45:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 8:45:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260807 08:45:09.014956 29106 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.319s user 0.005s sys 0.163s
I20260807 08:45:09.015084 29106 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 08:45:09.015136 29106 mini_ranger_kms.cc:401] 127.28.108.148:44011/kms/v1/keys
Aug 07 08:45:09 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786092309, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.108.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 08:45:09 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092309, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.108.148@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
Aug 07 08:45:10 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092309, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.108.148@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
Aug 07 08:45:12 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:45:12.731011 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34235
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:34235
--ranger_config_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:44011/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:12.832633 9875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:12.832888 9875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:12.832916 9875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:12.834510 9875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:45:12.834611 9875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:12.834635 9875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:12.834650 9875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:45:12.834664 9875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:45:12.836750 9875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.108.148:44011/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:34235
--ranger_config_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:12.837085 9875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:12.837416 9875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:12.843976 9883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:12.844029 9880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:12.844244 9875 server_base.cc:1034] running on GCE node
W20260807 08:45:12.844233 9881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:12.844792 9875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:12.845233 9875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:12.846468 9875 hybrid_clock.cc:648] HybridClock initialized: now 1786092312846439 us; error 45 us; skew 500 ppm
Aug 07 08:45:12 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092312, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:12.849993 9875 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:45:12.850688 9875 webserver.cc:460] Webserver started at http://127.28.108.190:38045/ using document root <none> and password file <none>
I20260807 08:45:12.851011 9875 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:12.851060 9875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:12.851258 9875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 08:45:12 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092312, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:45:12.920490 9875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "88e84ba9332b470db087de11711b0ca8"
format_stamp: "Formatted at 2026-08-07 08:45:12 on dist-test-slave-9zdj"
server_key: "9c25ae92d43def1d620f2aed0143cac0"
server_key_iv: "c022f8eb0ecf2dc4bc60612d17f28776"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:12.920982 9875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "88e84ba9332b470db087de11711b0ca8"
format_stamp: "Formatted at 2026-08-07 08:45:12 on dist-test-slave-9zdj"
server_key: "9c25ae92d43def1d620f2aed0143cac0"
server_key_iv: "c022f8eb0ecf2dc4bc60612d17f28776"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:12.922775 9875 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 08:45:12.939312 9890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:12.939610 9875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:45:12.939738 9875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "88e84ba9332b470db087de11711b0ca8"
format_stamp: "Formatted at 2026-08-07 08:45:12 on dist-test-slave-9zdj"
server_key: "9c25ae92d43def1d620f2aed0143cac0"
server_key_iv: "c022f8eb0ecf2dc4bc60612d17f28776"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:12.939841 9875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:12.960942 9875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:12.963435 9875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:12.963634 9875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:12.969408 9875 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:34235
I20260807 08:45:12.970059 9947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:34235 every 8 connection(s)
I20260807 08:45:12.970109 9875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:45:12.971084 9948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:12.973730 9948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8: Bootstrap starting.
I20260807 08:45:12.974468 9948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:12.974807 9948 log.cc:824] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:12.975633 9948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8: No bootstrap required, opened a new log
I20260807 08:45:12.977932 9948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } }
I20260807 08:45:12.978112 9948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:12.978138 9948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e84ba9332b470db087de11711b0ca8, State: Initialized, Role: FOLLOWER
I20260807 08:45:12.978313 9948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } }
I20260807 08:45:12.978404 9948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:45:12.978451 9948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:45:12.978503 9948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:12.979132 9948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } }
I20260807 08:45:12.979236 9948 leader_election.cc:302] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e84ba9332b470db087de11711b0ca8; no voters:
I20260807 08:45:12.979511 9948 leader_election.cc:288] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:45:12.979545 9951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:45:12.979671 9951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 1 LEADER]: Becoming Leader. State: Replica: 88e84ba9332b470db087de11711b0ca8, State: Running, Role: LEADER
I20260807 08:45:12.979820 9948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:45:12.979831 9951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } }
I20260807 08:45:12.980307 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 9875
I20260807 08:45:12.980419 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:45:12.980618 29106 external_mini_cluster.cc:1428] Setting key b60f84b8fe17c537482500c72b69e0ea
I20260807 08:45:12.980549 9953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e84ba9332b470db087de11711b0ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } } }
I20260807 08:45:12.981132 9952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e84ba9332b470db087de11711b0ca8. Latest consensus state: current_term: 1 leader_uuid: "88e84ba9332b470db087de11711b0ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } } }
I20260807 08:45:12.981244 9952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:12.980767 9953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:12.981755 9948 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 08:45:12.985388 9968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:45:12.987326 9968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 07 08:45:12 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:12.995838 9968 catalog_manager.cc:1347] Generated new cluster ID: fa254a2949054f53b38e7fc8de90d974
I20260807 08:45:12.996016 9968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:45:12.997471 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:12.981523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.28.108.190:34235)
0807 08:45:12.981741 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:12.981754 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:12.981976 (+ 222us) server_negotiation.cc:407] Connection header received
0807 08:45:12.982126 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:12.982135 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:12.982204 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:12.983974 (+ 1770us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:12.984536 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:12.986711 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:12.988008 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:12.988676 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:12.988811 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:12.992155 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:12.992160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:12.992161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:12.992177 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:12.994721 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:12.995662 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:12.995665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:12.995666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:12.995798 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:12.996057 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:12.996057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:12.996058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:12.996418 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:12.997037 (+ 619us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:12.997110 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 08:45:12.997247 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
I20260807 08:45:13.021965 9968 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:45:13.022897 9968 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:45:13.036008 9968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8: Generated new TSK 0
I20260807 08:45:13.036530 9968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:45:13.861970 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:44011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34235
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:13.993963 10000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:13.994233 10000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:13.994279 10000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:13.995672 10000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:13.995769 10000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:13.995818 10000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:45:13.997665 10000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.108.148:44011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:13.997999 10000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:13.998263 10000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:14.004083 10005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:14.004086 10006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:14.004413 10008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:14.004506 10000 server_base.cc:1034] running on GCE node
I20260807 08:45:14.004828 10000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:14.005170 10000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:14.006377 10000 hybrid_clock.cc:648] HybridClock initialized: now 1786092314006361 us; error 29 us; skew 500 ppm
Aug 07 08:45:14 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:14.019994 10000 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:45:14.020663 10000 webserver.cc:460] Webserver started at http://127.28.108.129:34089/ using document root <none> and password file <none>
I20260807 08:45:14.020967 10000 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:14.021046 10000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:14.021227 10000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 08:45:14 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:45:14.041026 10000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "f3ac229a95bf4cafaba94c03fcc6fb24"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "cc558387898315931a97fcebb20ee11c"
server_key_iv: "e8b00217a1f8f0fe2e150fa0aabf90f1"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:14.041678 10000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "f3ac229a95bf4cafaba94c03fcc6fb24"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "cc558387898315931a97fcebb20ee11c"
server_key_iv: "e8b00217a1f8f0fe2e150fa0aabf90f1"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:14.043825 10000 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260807 08:45:14.061877 10018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:14.062173 10000 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260807 08:45:14.062288 10000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "f3ac229a95bf4cafaba94c03fcc6fb24"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "cc558387898315931a97fcebb20ee11c"
server_key_iv: "e8b00217a1f8f0fe2e150fa0aabf90f1"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:14.062395 10000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:14.075373 10000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:14.077806 10000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:14.078071 10000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:14.078377 10000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:14.078855 10000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:14.078979 10000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:14.079087 10000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:14.079170 10000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:14.087514 10000 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:45845
I20260807 08:45:14.088038 10000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:45:14.088768 10139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:45845 every 8 connection(s)
I20260807 08:45:14.094425 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 10000
I20260807 08:45:14.094553 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:45:14.094786 29106 external_mini_cluster.cc:1428] Setting key e67fa9ada3a93fb930bdd6c19824cb36
Aug 07 08:45:14 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:14.101279 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:14.088910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34235 (local address 127.28.108.129:45325)
0807 08:45:14.089106 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:14.089155 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:14.089872 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:14.090263 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:14.090268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:14.090544 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:14.090867 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:14.090874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.091957 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.091958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:14.093402 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:14.093406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.094027 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.094028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:14.094195 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:14.094904 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:14.094916 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:14.096937 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:14.099113 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:14.099127 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:14.099130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:14.099427 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:14.099715 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:14.099716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:14.099716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:14.099951 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:14.100303 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:14.100313 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:14.100537 (+ 224us) client_negotiation.cc:769] Sending connection context
0807 08:45:14.100660 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 08:45:14.100951 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":118,"threads_started":1}
I20260807 08:45:14.101820 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:14.089323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:45325 (local address 127.28.108.190:34235)
0807 08:45:14.089526 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:14.089529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:14.089546 (+ 17us) server_negotiation.cc:407] Connection header received
0807 08:45:14.090044 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:14.090046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:14.090060 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:14.090146 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:14.090998 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.091831 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.093529 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.093891 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.093981 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:14.097106 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:14.097107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:14.097107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:14.097115 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:14.098998 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:14.099526 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:14.099527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:14.099528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:14.099633 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:14.100032 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:14.100033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:14.100034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:14.100208 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:14.100270 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:14.101459 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0807 08:45:14.101649 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260807 08:45:14.102579 10140 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34235
I20260807 08:45:14.102733 10140 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:14.103003 10140 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
I20260807 08:45:14.103771 9910 ts_manager.cc:194] Registered new tserver with Master: f3ac229a95bf4cafaba94c03fcc6fb24 (127.28.108.129:45845)
I20260807 08:45:14.104811 9910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:45325
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:45:14.137387 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:44011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34235
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 08:45:14 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 08:45:14.273921 10146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:14.274432 10146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:14.274550 10146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:14.276760 10146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:14.276927 10146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:14.277020 10146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:45:14.279718 10146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.108.148:44011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:14.280357 10146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:14.280684 10146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:14.288520 10155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:14.288659 10157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:14.288518 10159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:14.289590 10146 server_base.cc:1034] running on GCE node
I20260807 08:45:14.289973 10146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:14.290326 10146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:14.291517 10146 hybrid_clock.cc:648] HybridClock initialized: now 1786092314291493 us; error 41 us; skew 500 ppm
Aug 07 08:45:14 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:14.295398 10146 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:45:14.296020 10146 webserver.cc:460] Webserver started at http://127.28.108.130:45649/ using document root <none> and password file <none>
I20260807 08:45:14.296362 10146 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:14.296433 10146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:14.296586 10146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 08:45:14 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:45:14.312839 10146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "b31531a734f7464788ba7fe48d2d8363"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "9ce2e0e2194b0d690f8645ebaf7ab3b3"
server_key_iv: "904e10a2d77aecf8d717d83e1df5df6d"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:14.313380 10146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "b31531a734f7464788ba7fe48d2d8363"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "9ce2e0e2194b0d690f8645ebaf7ab3b3"
server_key_iv: "904e10a2d77aecf8d717d83e1df5df6d"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:14.315505 10146 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 08:45:14.330327 10166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:14.330677 10146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:45:14.330794 10146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "b31531a734f7464788ba7fe48d2d8363"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "9ce2e0e2194b0d690f8645ebaf7ab3b3"
server_key_iv: "904e10a2d77aecf8d717d83e1df5df6d"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:14.330902 10146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:14.368330 10146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:14.371377 10146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:14.371565 10146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:14.371896 10146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:14.372339 10146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:14.372396 10146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:14.372430 10146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:14.372449 10146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:14.379880 10146 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:42723
I20260807 08:45:14.380520 10146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:45:14.381157 10290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:42723 every 8 connection(s)
I20260807 08:45:14.390352 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 10146
I20260807 08:45:14.390480 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
Aug 07 08:45:14 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:14.390708 29106 external_mini_cluster.cc:1428] Setting key b6c8cac83361274325ac6fc185509999
I20260807 08:45:14.397981 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:14.382168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:55301 (local address 127.28.108.190:34235)
0807 08:45:14.382231 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:14.382233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:14.382795 (+ 562us) server_negotiation.cc:407] Connection header received
0807 08:45:14.383600 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:14.383601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:14.383622 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:14.383657 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:14.384710 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.385724 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.387640 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.387977 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.388044 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:14.393813 (+ 5769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:14.393815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:14.393815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:14.393828 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:14.395705 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:14.396354 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:14.396355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:14.396356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:14.396426 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:14.396915 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:14.396917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:14.396917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:14.397107 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:14.397177 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:14.397764 (+ 587us) server_negotiation.cc:299] Negotiation successful
0807 08:45:14.397882 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260807 08:45:14.398869 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:14.382148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34235 (local address 127.28.108.130:55301)
0807 08:45:14.382635 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:14.382683 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:14.383450 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:14.383876 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:14.383880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:14.384194 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:14.384528 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:14.384536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.385880 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.385881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:14.387445 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:14.387449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.388162 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.388164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:14.388343 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:14.389142 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:14.389153 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:14.391392 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:14.395838 (+ 4446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:14.395852 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:14.395856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:14.396237 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:14.396503 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:14.396504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:14.396504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:14.396678 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:14.397254 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:14.397264 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:14.397473 (+ 209us) client_negotiation.cc:769] Sending connection context
0807 08:45:14.398370 (+ 897us) client_negotiation.cc:240] Negotiation successful
0807 08:45:14.398582 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":123,"threads_started":1}
I20260807 08:45:14.399794 10292 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34235
I20260807 08:45:14.399909 10292 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:14.400117 10292 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
I20260807 08:45:14.400592 9910 ts_manager.cc:194] Registered new tserver with Master: b31531a734f7464788ba7fe48d2d8363 (127.28.108.130:42723)
I20260807 08:45:14.401171 9910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:55301
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:45:14.442018 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:44011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34235
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:14.611624 10298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:14.612051 10298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:14.612134 10298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:14.614130 10298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:14.614274 10298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:14.614349 10298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:45:14.616838 10298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.108.148:44011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:34235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:14.617390 10298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:14.617794 10298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:14.624917 10304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:14.624929 10303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:14.625419 10306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:14.626624 10298 server_base.cc:1034] running on GCE node
I20260807 08:45:14.626988 10298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:14.627283 10298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:14.630622 10298 hybrid_clock.cc:648] HybridClock initialized: now 1786092314630579 us; error 45 us; skew 500 ppm
Aug 07 08:45:14 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:14.634554 10298 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:45:14.635272 10298 webserver.cc:460] Webserver started at http://127.28.108.131:41437/ using document root <none> and password file <none>
I20260807 08:45:14.635839 10298 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:14.635918 10298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:14.636080 10298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 08:45:14 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:45:14.656908 10298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "521322366bbc4227b01be808d4941264"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "bff5798aa6ba8f84d4e02e447f7aed90"
server_key_iv: "dd35603cc5978b80b404f741e76a113e"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:14.657532 10298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "521322366bbc4227b01be808d4941264"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "bff5798aa6ba8f84d4e02e447f7aed90"
server_key_iv: "dd35603cc5978b80b404f741e76a113e"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:14.661054 10298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260807 08:45:14.683414 10314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:14.683715 10298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:45:14.683820 10298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "521322366bbc4227b01be808d4941264"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "bff5798aa6ba8f84d4e02e447f7aed90"
server_key_iv: "dd35603cc5978b80b404f741e76a113e"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:14.683925 10298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:14.717468 10298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:14.720623 10298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:14.720933 10298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:14.721271 10298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:14.721822 10298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:14.721923 10298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:14.722007 10298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:14.722064 10298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:14.729175 10298 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:36431
I20260807 08:45:14.732275 10434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:36431 every 8 connection(s)
I20260807 08:45:14.732986 10298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
Aug 07 08:45:14 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:14.741626 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 10298
I20260807 08:45:14.741762 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:45:14.741992 29106 external_mini_cluster.cc:1428] Setting key 95df53a08c90a5aefeca046e5550c7ba
I20260807 08:45:14.746395 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:14.734213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:50571 (local address 127.28.108.190:34235)
0807 08:45:14.734268 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:14.734271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:14.734684 (+ 413us) server_negotiation.cc:407] Connection header received
0807 08:45:14.735388 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:14.735390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:14.735403 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:14.735447 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:14.736256 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.736868 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.738472 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.738711 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.738804 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:14.742278 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:14.742281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:14.742281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:14.742292 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:14.744242 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:14.744891 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:14.744893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:14.744893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:14.744967 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:14.745425 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:14.745426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:14.745427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:14.745592 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:14.745703 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:14.746161 (+ 458us) server_negotiation.cc:299] Negotiation successful
0807 08:45:14.746267 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 08:45:14.747311 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:14.734224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34235 (local address 127.28.108.131:50571)
0807 08:45:14.734530 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:14.734578 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:14.735277 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:14.735539 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:14.735543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:14.735842 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:14.736150 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:14.736157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.736988 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.736989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:14.738368 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:14.738372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:14.738830 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:14.738832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:14.738987 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:14.739740 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:14.739753 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:14.741856 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:14.744376 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:14.744391 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:14.744394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:14.744786 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:14.745060 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:14.745062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:14.745062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:14.745326 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:14.745761 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:14.745770 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:14.745982 (+ 212us) client_negotiation.cc:769] Sending connection context
0807 08:45:14.746826 (+ 844us) client_negotiation.cc:240] Negotiation successful
0807 08:45:14.747042 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260807 08:45:14.748070 10435 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34235
I20260807 08:45:14.748183 10435 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:14.748405 10435 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
I20260807 08:45:14.748970 9910 ts_manager.cc:194] Registered new tserver with Master: 521322366bbc4227b01be808d4941264 (127.28.108.131:36431)
I20260807 08:45:14.749480 9910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:50571
I20260807 08:45:14.753973 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:45:14.754912 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 10000
I20260807 08:45:14.787112 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 10146
I20260807 08:45:14.797453 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 10298
I20260807 08:45:14.808722 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 9875
I20260807 08:45:14.820616 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34235
--webserver_interface=127.28.108.190
--webserver_port=38045
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:34235
--ranger_config_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:44011/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.108.148:44011/kms with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:14.916186 10438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:14.916450 10438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:14.916483 10438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:14.917801 10438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:45:14.917852 10438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:14.917868 10438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:14.917881 10438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:45:14.917893 10438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:45:14.919736 10438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.108.148:44011/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:34235
--ranger_config_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=38045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:14.920045 10438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:14.920452 10438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:14.926568 10445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:14.926697 10446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:14.926851 10448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:14.927233 10438 server_base.cc:1034] running on GCE node
I20260807 08:45:14.927506 10438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:14.927861 10438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:14.929059 10438 hybrid_clock.cc:648] HybridClock initialized: now 1786092314929035 us; error 28 us; skew 500 ppm
Aug 07 08:45:14 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:14.932672 10438 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:45:14.933349 10438 webserver.cc:460] Webserver started at http://127.28.108.190:38045/ using document root <none> and password file <none>
I20260807 08:45:14.933729 10438 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:14.933856 10438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 08:45:15 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:45:15.018755 10438 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:45:15.019778 10457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:15.020061 10438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:15.020185 10438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "88e84ba9332b470db087de11711b0ca8"
format_stamp: "Formatted at 2026-08-07 08:45:12 on dist-test-slave-9zdj"
server_key: "9c25ae92d43def1d620f2aed0143cac0"
server_key_iv: "c022f8eb0ecf2dc4bc60612d17f28776"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:15.020715 10438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:15.058727 10438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:15.061743 10438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:15.061959 10438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:15.066913 10438 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:34235
I20260807 08:45:15.066963 10512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:34235 every 8 connection(s)
I20260807 08:45:15.067428 10438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:45:15.068953 10513 sys_catalog.cc:263] Verifying existing consensus state
I20260807 08:45:15.069782 10513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8: Bootstrap starting.
I20260807 08:45:15.071372 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 10438
I20260807 08:45:15.071789 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:44011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:45845
--local_ip_for_outbound_sockets=127.28.108.129
--tserver_master_addrs=127.28.108.190:34235
--webserver_port=34089
--webserver_interface=127.28.108.129
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.108.148:44011/kms with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:45:15.079337 10513 log.cc:824] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:15.081943 10513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:45:15.082335 10513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8: Bootstrap complete.
I20260807 08:45:15.085224 10513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } }
I20260807 08:45:15.085546 10513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e84ba9332b470db087de11711b0ca8, State: Initialized, Role: FOLLOWER
I20260807 08:45:15.085796 10513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } }
I20260807 08:45:15.085896 10513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:45:15.085961 10513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:45:15.086019 10513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:45:15.087442 10513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } }
I20260807 08:45:15.087589 10513 leader_election.cc:302] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e84ba9332b470db087de11711b0ca8; no voters:
I20260807 08:45:15.087843 10513 leader_election.cc:288] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 08:45:15.087888 10517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:45:15.088011 10517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [term 2 LEADER]: Becoming Leader. State: Replica: 88e84ba9332b470db087de11711b0ca8, State: Running, Role: LEADER
I20260807 08:45:15.088188 10513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:45:15.088209 10517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } }
I20260807 08:45:15.088727 10519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "88e84ba9332b470db087de11711b0ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } } }
I20260807 08:45:15.088754 10518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e84ba9332b470db087de11711b0ca8. Latest consensus state: current_term: 2 leader_uuid: "88e84ba9332b470db087de11711b0ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e84ba9332b470db087de11711b0ca8" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34235 } } }
I20260807 08:45:15.088973 10518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:15.089254 10519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:15.089450 10526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:45:15.090224 10526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:45:15.090461 10526 catalog_manager.cc:1259] Loaded cluster ID: fa254a2949054f53b38e7fc8de90d974
I20260807 08:45:15.090559 10526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:45:15.090677 10513 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 08:45:15.095760 10526 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:45:15.096590 10526 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8: Loaded TSK: 0
I20260807 08:45:15.097023 10526 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 08:45:15.207477 10515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:15.207805 10515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:15.207870 10515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:15.209781 10515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:15.209883 10515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:15.209941 10515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:45:15.212121 10515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.108.148:44011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:45845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=34089
--tserver_master_addrs=127.28.108.190:34235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:15.212502 10515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:15.212795 10515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:15.220553 10554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:15.220853 10515 server_base.cc:1034] running on GCE node
W20260807 08:45:15.223556 10556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:15.223995 10553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:15.224447 10515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:15.224836 10515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:15.226045 10515 hybrid_clock.cc:648] HybridClock initialized: now 1786092315226025 us; error 72 us; skew 500 ppm
Aug 07 08:45:15 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092315, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:15.229727 10515 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:45:15.230486 10515 webserver.cc:460] Webserver started at http://127.28.108.129:34089/ using document root <none> and password file <none>
I20260807 08:45:15.230796 10515 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:15.230902 10515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 08:45:15 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092315, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:45:15.247056 10515 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:45:15.248922 10564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:15.249226 10515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:45:15.249337 10515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "f3ac229a95bf4cafaba94c03fcc6fb24"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "cc558387898315931a97fcebb20ee11c"
server_key_iv: "e8b00217a1f8f0fe2e150fa0aabf90f1"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:15.249931 10515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:15.268162 10515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:15.271067 10515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:15.271255 10515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:15.271507 10515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:15.271890 10515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:15.271937 10515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:15.271970 10515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:15.271996 10515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:15.279852 10515 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:45845
I20260807 08:45:15.280459 10515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:45:15.284019 10686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:45845 every 8 connection(s)
I20260807 08:45:15.288508 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 10515
I20260807 08:45:15.288904 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:44011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:42723
--local_ip_for_outbound_sockets=127.28.108.130
--tserver_master_addrs=127.28.108.190:34235
--webserver_port=45649
--webserver_interface=127.28.108.130
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.108.148:44011/kms with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 08:45:15 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092315, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:15.303997 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:15.281522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34235 (local address 127.28.108.129:49841)
0807 08:45:15.282058 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:15.282115 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:15.282984 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:15.283653 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:15.283658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:15.283968 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:15.284318 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:15.284326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.286427 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.286428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:15.288104 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:15.288109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.291150 (+ 3041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.291152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:15.291371 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:15.292150 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:15.292163 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:15.298124 (+ 5961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:15.301440 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:15.301454 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:15.301457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:15.301911 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:15.302294 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:15.302295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:15.302295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:15.302461 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:15.303139 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:15.303147 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:15.303339 (+ 192us) client_negotiation.cc:769] Sending connection context
0807 08:45:15.303454 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 08:45:15.303667 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":146,"threads_started":1}
I20260807 08:45:15.304899 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:15.281687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:49841 (local address 127.28.108.190:34235)
0807 08:45:15.281949 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:15.281960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:15.282419 (+ 459us) server_negotiation.cc:407] Connection header received
0807 08:45:15.283301 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:15.283310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:15.283383 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:15.283561 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:15.284488 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.286253 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.288242 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.289598 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.289784 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:15.298355 (+ 8571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:15.298362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:15.298363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:15.298386 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:15.301284 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:15.302042 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:15.302046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:15.302047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:15.302203 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:15.302551 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:15.302552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:15.302552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:15.302978 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:15.303099 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:15.303873 (+ 774us) server_negotiation.cc:299] Negotiation successful
0807 08:45:15.304641 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":105,"threads_started":1}
I20260807 08:45:15.305800 10687 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34235
I20260807 08:45:15.305932 10687 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:15.306159 10687 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
W20260807 08:45:15.307451 10687 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:34235 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 08:45:15.307551 10687 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:15.307633 10687 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
I20260807 08:45:15.308262 10687 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:15.308365 10687 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
W20260807 08:45:15.308807 10687 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 08:45:15.410537 10692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:15.410796 10692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:15.410826 10692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:15.412456 10692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:15.412524 10692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:15.412560 10692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:45:15.415457 10692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.108.148:44011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:42723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=45649
--tserver_master_addrs=127.28.108.190:34235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:15.415800 10692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:15.416168 10692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:15.424837 10700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:15.424794 10698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:15.424916 10697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:15.425514 10692 server_base.cc:1034] running on GCE node
I20260807 08:45:15.425899 10692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:15.426474 10692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:15.427695 10692 hybrid_clock.cc:648] HybridClock initialized: now 1786092315427677 us; error 32 us; skew 500 ppm
Aug 07 08:45:15 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092315, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:15.432045 10692 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:45:15.432812 10692 webserver.cc:460] Webserver started at http://127.28.108.130:45649/ using document root <none> and password file <none>
I20260807 08:45:15.433190 10692 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:15.433310 10692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 08:45:15 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092315, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:45:15.455448 10692 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260807 08:45:15.458242 10709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:15.458506 10692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 08:45:15.458627 10692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "b31531a734f7464788ba7fe48d2d8363"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "9ce2e0e2194b0d690f8645ebaf7ab3b3"
server_key_iv: "904e10a2d77aecf8d717d83e1df5df6d"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:15.459196 10692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:15.514051 10692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:15.516983 10692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:15.517182 10692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:15.517517 10692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:15.517989 10692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:15.518050 10692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:15.518085 10692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:15.518105 10692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:15.526158 10692 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:42723
I20260807 08:45:15.526221 10832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:42723 every 8 connection(s)
I20260807 08:45:15.526661 10692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:45:15.530262 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 10692
I20260807 08:45:15.530669 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.108.148:44011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:36431
--local_ip_for_outbound_sockets=127.28.108.131
--tserver_master_addrs=127.28.108.190:34235
--webserver_port=41437
--webserver_interface=127.28.108.131
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.108.148:44011/kms with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 08:45:15 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092315, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:15.542923 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:15.527725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:48271 (local address 127.28.108.190:34235)
0807 08:45:15.527806 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:15.527809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:15.528037 (+ 228us) server_negotiation.cc:407] Connection header received
0807 08:45:15.528590 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:15.528592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:15.528608 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:15.528668 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:15.529683 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.530585 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.533017 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.533424 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.533482 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:15.537904 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:15.537908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:15.537908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:15.537921 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:15.540530 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:15.541290 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:15.541292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:15.541293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:15.541371 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:15.541839 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:15.541841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:15.541842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:15.542036 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:15.542092 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:15.542613 (+ 521us) server_negotiation.cc:299] Negotiation successful
0807 08:45:15.542756 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 08:45:15.543797 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:15.527722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34235 (local address 127.28.108.130:48271)
0807 08:45:15.527914 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:15.527959 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:15.528471 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:15.528752 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:15.528759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:15.529061 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:15.529463 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:15.529501 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.531012 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.531014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:15.532834 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:15.532840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.533670 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.533672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:15.533886 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:15.534758 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:15.534772 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:15.537714 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:15.540695 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:15.540710 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:15.540714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:15.541171 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:15.541462 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:15.541463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:15.541463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:15.541684 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:15.542154 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:15.542164 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:15.542410 (+ 246us) client_negotiation.cc:769] Sending connection context
0807 08:45:15.543208 (+ 798us) client_negotiation.cc:240] Negotiation successful
0807 08:45:15.543455 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":118,"threads_started":1}
I20260807 08:45:15.544543 10834 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34235
I20260807 08:45:15.544734 10834 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:15.545061 10834 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
W20260807 08:45:15.546066 10834 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:34235 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 08:45:15.546209 10834 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:15.546332 10834 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
I20260807 08:45:15.546741 10834 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:15.546878 10834 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
W20260807 08:45:15.547264 10834 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 08:45:15.649991 10837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:15.650302 10837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:15.650331 10837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:15.651717 10837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:15.651782 10837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:15.651813 10837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:45:15.653578 10837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.108.148:44011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:36431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=41437
--tserver_master_addrs=127.28.108.190:34235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:15.653975 10837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:15.654268 10837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:15.660697 10848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:15.661047 10845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:15.661242 10837 server_base.cc:1034] running on GCE node
W20260807 08:45:15.661875 10843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:15.662192 10837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:15.662588 10837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:15.663766 10837 hybrid_clock.cc:648] HybridClock initialized: now 1786092315663749 us; error 31 us; skew 500 ppm
Aug 07 08:45:15 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092315, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:15.667469 10837 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:45:15.667999 10837 webserver.cc:460] Webserver started at http://127.28.108.131:41437/ using document root <none> and password file <none>
I20260807 08:45:15.668296 10837 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:15.668393 10837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 08:45:15 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092315, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:45:15.702005 10837 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:15.703354 10857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:15.703598 10837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:15.703708 10837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "521322366bbc4227b01be808d4941264"
format_stamp: "Formatted at 2026-08-07 08:45:14 on dist-test-slave-9zdj"
server_key: "bff5798aa6ba8f84d4e02e447f7aed90"
server_key_iv: "dd35603cc5978b80b404f741e76a113e"
server_key_version: "kuduclusterkey@0"
I20260807 08:45:15.704272 10837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:15.731490 10837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:15.734344 10837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:15.734558 10837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:15.734889 10837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:15.735309 10837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:15.735358 10837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:15.735390 10837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:15.735407 10837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:15.743027 10837 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:36431
I20260807 08:45:15.743561 10837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:45:15.743726 10978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:36431 every 8 connection(s)
Aug 07 08:45:15 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092315, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:15.751982 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 10837
I20260807 08:45:15.757742 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:15.744661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:44781 (local address 127.28.108.190:34235)
0807 08:45:15.744716 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:15.744719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:15.745095 (+ 376us) server_negotiation.cc:407] Connection header received
0807 08:45:15.745930 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:15.745932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:15.745949 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:15.746013 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:15.746787 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.747376 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.748764 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.749027 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.749143 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:15.752976 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:15.752978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:15.752978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:15.752987 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:15.755601 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:15.756305 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:15.756307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:15.756308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:15.756388 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:15.756746 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:15.756747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:15.756748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:15.757009 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:15.757066 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:15.757498 (+ 432us) server_negotiation.cc:299] Negotiation successful
0807 08:45:15.757600 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 08:45:15.758399 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:15.744691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:34235 (local address 127.28.108.131:44781)
0807 08:45:15.744917 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:15.744966 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:15.745764 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:15.746104 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:15.746109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:15.746407 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:15.746668 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:15.746674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.747522 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.747524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:15.748644 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:15.748647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.749196 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.749198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:15.749349 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:15.750108 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:15.750120 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:15.752853 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:15.755732 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:15.755746 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:15.755749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:15.756197 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:15.756474 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:15.756475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:15.756475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:15.756649 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:15.757121 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:15.757129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:15.757329 (+ 200us) client_negotiation.cc:769] Sending connection context
0807 08:45:15.757924 (+ 595us) client_negotiation.cc:240] Negotiation successful
0807 08:45:15.758140 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":140,"threads_started":1}
I20260807 08:45:15.759002 10979 heartbeater.cc:347] Connected to a master server at 127.28.108.190:34235
I20260807 08:45:15.759166 10979 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:15.759388 10979 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
W20260807 08:45:15.760123 10979 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:34235 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 08:45:15.760231 10979 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:15.760325 10979 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
I20260807 08:45:15.760716 10979 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:15.760820 10979 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
W20260807 08:45:15.761173 10979 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 08:45:15.766897 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:15.752534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.28.108.190:34235)
0807 08:45:15.752711 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:15.752713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:15.752858 (+ 145us) server_negotiation.cc:407] Connection header received
0807 08:45:15.752973 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:15.752975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:15.752990 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:15.753039 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:15.755412 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.756249 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.759696 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:15.760472 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:15.760524 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:15.763049 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:15.763050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:15.763051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:15.763060 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:15.765182 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:15.765739 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:15.765741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:15.765741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:15.765819 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:15.766054 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:15.766055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:15.766056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:15.766208 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:15.766399 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:15.766632 (+ 233us) server_negotiation.cc:299] Negotiation successful
0807 08:45:15.766740 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260807 08:45:16.309032 10687 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:16.309267 10687 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
I20260807 08:45:16.309754 10475 ts_manager.cc:194] Registered new tserver with Master: f3ac229a95bf4cafaba94c03fcc6fb24 (127.28.108.129:45845)
I20260807 08:45:16.310931 10475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:49841
Aug 07 08:45:16 dist-test-slave-9zdj krb5kdc[9380](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786092316, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:16.547493 10834 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:16.547688 10834 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
I20260807 08:45:16.548197 10475 ts_manager.cc:194] Registered new tserver with Master: b31531a734f7464788ba7fe48d2d8363 (127.28.108.130:42723)
I20260807 08:45:16.548849 10475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:48271
I20260807 08:45:16.761404 10979 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:16.761600 10979 heartbeater.cc:511] Master 127.28.108.190:34235 requested a full tablet report, sending...
I20260807 08:45:16.762065 10475 ts_manager.cc:194] Registered new tserver with Master: 521322366bbc4227b01be808d4941264 (127.28.108.131:36431)
I20260807 08:45:16.762494 10475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:44781
I20260807 08:45:16.764144 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:45:16.774348 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.765290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.28.108.190:34235)
0807 08:45:16.765611 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:16.765613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:16.765620 (+ 7us) server_negotiation.cc:407] Connection header received
0807 08:45:16.765662 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:16.765664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:16.765676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:16.765745 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:16.766524 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.767313 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.768460 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.768760 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.768863 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.770510 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:16.770512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:16.770512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:16.770521 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:16.772527 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:16.773003 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:16.773005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:16.773005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:16.773078 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:16.773303 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:16.773303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:16.773304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:16.773470 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:16.773574 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:16.774128 (+ 554us) server_negotiation.cc:299] Negotiation successful
0807 08:45:16.774231 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":65,"threads_started":1}
I20260807 08:45:16.777148 10475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:45:16.779755 10475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:45:16.795607 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.783503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:45845 (local address 127.0.0.1:36322)
0807 08:45:16.790251 (+ 6748us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:16.790258 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:16.790332 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:16.790678 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:16.790682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:16.790684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:16.790829 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:16.790832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.792274 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.792275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:16.793983 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:16.793987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.795189 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.795191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:16.795308 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.795331 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 08:45:16.795411 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 08:45:16.795491 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6738,"mutex_wait_us":13,"thread_start_us":68,"threads_started":1}
I20260807 08:45:16.796229 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.783664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.28.108.129:45845)
0807 08:45:16.783853 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:16.783859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:16.790349 (+ 6490us) server_negotiation.cc:407] Connection header received
0807 08:45:16.790413 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:16.790418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:16.790440 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:16.790531 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:16.790930 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.792154 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.794126 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.795064 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.795891 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.795995 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:16.796033 (+ 38us) server_negotiation.cc:299] Negotiation successful
0807 08:45:16.796087 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260807 08:45:16.796591 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.783236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:36431 (local address 127.0.0.1:40092)
0807 08:45:16.790146 (+ 6910us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:16.790155 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:16.790186 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:16.790678 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:16.790682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:16.790688 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:16.790829 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:16.790832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.792774 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.792775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:16.793985 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:16.793987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.795191 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.795192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:16.796294 (+ 1102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.796300 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:45:16.796458 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 08:45:16.796513 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6895,"mutex_wait_us":35,"thread_start_us":102,"threads_started":1}
I20260807 08:45:16.797237 10614 tablet_service.cc:1467] Processing CreateTablet for tablet 84cd134fdddc4d5cb5d4dd2b12f89e67 (DEFAULT_TABLE table=test-table [id=0ff26ce939ab4924870d653894b44d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:16.797797 10614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84cd134fdddc4d5cb5d4dd2b12f89e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:16.801543 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.783536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.28.108.131:36431)
0807 08:45:16.790474 (+ 6938us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:16.790481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:16.790492 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:45:16.790528 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:16.790530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:16.790547 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:16.790914 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:16.791009 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.792609 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.794125 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.795045 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.801212 (+ 6167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.801325 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:16.801364 (+ 39us) server_negotiation.cc:299] Negotiation successful
0807 08:45:16.801433 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6891,"thread_start_us":61,"threads_started":1}
I20260807 08:45:16.802389 10906 tablet_service.cc:1467] Processing CreateTablet for tablet 84cd134fdddc4d5cb5d4dd2b12f89e67 (DEFAULT_TABLE table=test-table [id=0ff26ce939ab4924870d653894b44d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:16.802734 11013 tablet_bootstrap.cc:492] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24: Bootstrap starting.
I20260807 08:45:16.802827 10906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84cd134fdddc4d5cb5d4dd2b12f89e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:16.805779 11013 tablet_bootstrap.cc:654] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:16.806288 11013 log.cc:824] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:16.807312 11013 tablet_bootstrap.cc:492] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24: No bootstrap required, opened a new log
I20260807 08:45:16.807397 11013 ts_tablet_manager.cc:1397] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260807 08:45:16.808601 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.783235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:42723 (local address 127.0.0.1:59490)
0807 08:45:16.789866 (+ 6631us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:16.789943 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:16.790039 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:16.796041 (+ 6002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:16.796042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:16.796045 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:16.796128 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:16.796130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.798102 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.798103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:16.799409 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:16.799414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.808264 (+ 8850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.808266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:16.808381 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.808390 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 08:45:16.808436 (+ 46us) client_negotiation.cc:240] Negotiation successful
0807 08:45:16.808504 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6574,"thread_start_us":104,"threads_started":1}
I20260807 08:45:16.809219 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.783707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.28.108.130:42723)
0807 08:45:16.795182 (+ 11475us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:16.795189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:16.795208 (+ 19us) server_negotiation.cc:407] Connection header received
0807 08:45:16.795257 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:16.795259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:16.795296 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:16.795419 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:16.796927 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.797962 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.799566 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.801057 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.808833 (+ 7776us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.808991 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:16.809030 (+ 39us) server_negotiation.cc:299] Negotiation successful
0807 08:45:16.809127 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11422,"thread_start_us":111,"threads_started":1}
I20260807 08:45:16.809985 11013 raft_consensus.cc:348] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31531a734f7464788ba7fe48d2d8363" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42723 } } peers { permanent_uuid: "521322366bbc4227b01be808d4941264" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36431 } } peers { permanent_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 45845 } }
I20260807 08:45:16.810177 11013 raft_consensus.cc:374] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:16.809998 10760 tablet_service.cc:1467] Processing CreateTablet for tablet 84cd134fdddc4d5cb5d4dd2b12f89e67 (DEFAULT_TABLE table=test-table [id=0ff26ce939ab4924870d653894b44d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:16.810214 11013 raft_consensus.cc:729] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ac229a95bf4cafaba94c03fcc6fb24, State: Initialized, Role: FOLLOWER
I20260807 08:45:16.810463 10760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84cd134fdddc4d5cb5d4dd2b12f89e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:16.810886 11013 consensus_queue.cc:260] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31531a734f7464788ba7fe48d2d8363" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42723 } } peers { permanent_uuid: "521322366bbc4227b01be808d4941264" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36431 } } peers { permanent_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 45845 } }
I20260807 08:45:16.811609 11013 ts_tablet_manager.cc:1428] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260807 08:45:16.811892 10687 heartbeater.cc:503] Master 127.28.108.190:34235 was elected leader, sending a full tablet report...
I20260807 08:45:16.813807 11015 tablet_bootstrap.cc:492] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264: Bootstrap starting.
I20260807 08:45:16.815006 11015 tablet_bootstrap.cc:654] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:16.815480 11015 log.cc:824] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:16.816484 11015 tablet_bootstrap.cc:492] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264: No bootstrap required, opened a new log
I20260807 08:45:16.816583 11015 ts_tablet_manager.cc:1397] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:45:16.817854 11017 tablet_bootstrap.cc:492] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363: Bootstrap starting.
I20260807 08:45:16.818666 11017 tablet_bootstrap.cc:654] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:16.819039 11017 log.cc:824] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:16.820019 11017 tablet_bootstrap.cc:492] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363: No bootstrap required, opened a new log
I20260807 08:45:16.820089 11017 ts_tablet_manager.cc:1397] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:45:16.819929 11015 raft_consensus.cc:348] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31531a734f7464788ba7fe48d2d8363" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42723 } } peers { permanent_uuid: "521322366bbc4227b01be808d4941264" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36431 } } peers { permanent_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 45845 } }
I20260807 08:45:16.820163 11015 raft_consensus.cc:374] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:16.820254 11015 raft_consensus.cc:729] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521322366bbc4227b01be808d4941264, State: Initialized, Role: FOLLOWER
I20260807 08:45:16.820618 11015 consensus_queue.cc:260] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31531a734f7464788ba7fe48d2d8363" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42723 } } peers { permanent_uuid: "521322366bbc4227b01be808d4941264" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36431 } } peers { permanent_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 45845 } }
I20260807 08:45:16.821043 11015 ts_tablet_manager.cc:1428] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 08:45:16.821239 10979 heartbeater.cc:503] Master 127.28.108.190:34235 was elected leader, sending a full tablet report...
I20260807 08:45:16.822783 11017 raft_consensus.cc:348] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31531a734f7464788ba7fe48d2d8363" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42723 } } peers { permanent_uuid: "521322366bbc4227b01be808d4941264" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36431 } } peers { permanent_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 45845 } }
I20260807 08:45:16.823175 11017 raft_consensus.cc:374] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:16.823272 11017 raft_consensus.cc:729] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31531a734f7464788ba7fe48d2d8363, State: Initialized, Role: FOLLOWER
I20260807 08:45:16.823506 11017 consensus_queue.cc:260] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31531a734f7464788ba7fe48d2d8363" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42723 } } peers { permanent_uuid: "521322366bbc4227b01be808d4941264" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36431 } } peers { permanent_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 45845 } }
I20260807 08:45:16.823900 11017 ts_tablet_manager.cc:1428] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 08:45:16.824072 10834 heartbeater.cc:503] Master 127.28.108.190:34235 was elected leader, sending a full tablet report...
I20260807 08:45:16.849817 11016 raft_consensus.cc:482] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:45:16.849980 11016 raft_consensus.cc:504] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31531a734f7464788ba7fe48d2d8363" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42723 } } peers { permanent_uuid: "521322366bbc4227b01be808d4941264" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36431 } } peers { permanent_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 45845 } }
I20260807 08:45:16.850400 11016 leader_election.cc:288] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b31531a734f7464788ba7fe48d2d8363 (127.28.108.130:42723), 521322366bbc4227b01be808d4941264 (127.28.108.131:36431)
I20260807 08:45:16.857384 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.851676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:43847 (local address 127.28.108.131:36431)
0807 08:45:16.851756 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:16.851759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:16.852254 (+ 495us) server_negotiation.cc:407] Connection header received
0807 08:45:16.852353 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:16.852355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:16.852365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:16.852396 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:16.852651 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.853379 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.855361 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.856303 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.856369 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.856381 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:16.857107 (+ 726us) server_negotiation.cc:299] Negotiation successful
0807 08:45:16.857167 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 08:45:16.857924 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.851230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:36431 (local address 127.28.108.129:43847)
0807 08:45:16.852199 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:16.852209 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:16.852280 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:16.852467 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:16.852468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:16.852470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:16.852553 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:16.852556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.853507 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.853508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:16.855218 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:16.855224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.856469 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.856470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:16.856992 (+ 522us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.856998 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:45:16.857760 (+ 762us) client_negotiation.cc:240] Negotiation successful
0807 08:45:16.857816 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"thread_start_us":92,"threads_started":1}
I20260807 08:45:16.858441 10929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cd134fdddc4d5cb5d4dd2b12f89e67" candidate_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521322366bbc4227b01be808d4941264" is_pre_election: true
I20260807 08:45:16.858645 10929 raft_consensus.cc:2454] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3ac229a95bf4cafaba94c03fcc6fb24 in term 0.
I20260807 08:45:16.859019 10568 leader_election.cc:302] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 521322366bbc4227b01be808d4941264, f3ac229a95bf4cafaba94c03fcc6fb24; no voters:
I20260807 08:45:16.859175 11016 raft_consensus.cc:2781] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:45:16.859218 11016 raft_consensus.cc:482] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:45:16.859238 11016 raft_consensus.cc:3037] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:16.860222 11016 raft_consensus.cc:504] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31531a734f7464788ba7fe48d2d8363" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42723 } } peers { permanent_uuid: "521322366bbc4227b01be808d4941264" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36431 } } peers { permanent_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 45845 } }
I20260807 08:45:16.860378 11016 leader_election.cc:288] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [CANDIDATE]: Term 1 election: Requested vote from peers b31531a734f7464788ba7fe48d2d8363 (127.28.108.130:42723), 521322366bbc4227b01be808d4941264 (127.28.108.131:36431)
I20260807 08:45:16.860750 10929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cd134fdddc4d5cb5d4dd2b12f89e67" candidate_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521322366bbc4227b01be808d4941264"
I20260807 08:45:16.860839 10929 raft_consensus.cc:3037] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:16.861742 10929 raft_consensus.cc:2454] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ac229a95bf4cafaba94c03fcc6fb24 in term 1.
I20260807 08:45:16.862010 10568 leader_election.cc:302] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 521322366bbc4227b01be808d4941264, f3ac229a95bf4cafaba94c03fcc6fb24; no voters:
I20260807 08:45:16.862141 11016 raft_consensus.cc:2781] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:45:16.862313 11016 raft_consensus.cc:686] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [term 1 LEADER]: Becoming Leader. State: Replica: f3ac229a95bf4cafaba94c03fcc6fb24, State: Running, Role: LEADER
I20260807 08:45:16.862412 11016 consensus_queue.cc:237] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31531a734f7464788ba7fe48d2d8363" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42723 } } peers { permanent_uuid: "521322366bbc4227b01be808d4941264" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36431 } } peers { permanent_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 45845 } }
I20260807 08:45:16.863391 10475 catalog_manager.cc:5495] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3ac229a95bf4cafaba94c03fcc6fb24 (127.28.108.129). New cstate: current_term: 1 leader_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31531a734f7464788ba7fe48d2d8363" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521322366bbc4227b01be808d4941264" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 45845 } health_report { overall_health: HEALTHY } } }
Aug 07 08:45:16 dist-test-slave-9zdj krb5kdc[9380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092316, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for HTTP/127.28.108.148@KRBTEST.COM
I20260807 08:45:16.866546 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.850717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:42723 (local address 127.28.108.129:51491)
0807 08:45:16.852006 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:16.852023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:16.852105 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:16.860929 (+ 8824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:16.860931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:16.860934 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:16.861018 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:16.861021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.862158 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.862159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:16.863494 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:16.863499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.865319 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.865321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:16.866259 (+ 938us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.866268 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 08:45:16.866343 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 08:45:16.866409 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"thread_start_us":111,"threads_started":1}
I20260807 08:45:16.867122 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.851077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:51491 (local address 127.28.108.130:42723)
0807 08:45:16.851820 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:16.851823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:16.852089 (+ 266us) server_negotiation.cc:407] Connection header received
0807 08:45:16.852168 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:16.852170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:16.852183 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:16.852261 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:16.861142 (+ 8881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.862022 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.863648 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.865090 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.865225 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.865241 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:16.866403 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0807 08:45:16.866517 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707}
I20260807 08:45:16.867575 10781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cd134fdddc4d5cb5d4dd2b12f89e67" candidate_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31531a734f7464788ba7fe48d2d8363" is_pre_election: true
I20260807 08:45:16.867780 10781 raft_consensus.cc:2454] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3ac229a95bf4cafaba94c03fcc6fb24 in term 0.
I20260807 08:45:16.867952 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cd134fdddc4d5cb5d4dd2b12f89e67" candidate_uuid: "f3ac229a95bf4cafaba94c03fcc6fb24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31531a734f7464788ba7fe48d2d8363"
I20260807 08:45:16.868028 10780 raft_consensus.cc:3037] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:16.869069 10780 raft_consensus.cc:2454] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ac229a95bf4cafaba94c03fcc6fb24 in term 1.
I20260807 08:45:16.885829 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.881834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.28.108.129:45845)
0807 08:45:16.881904 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:16.881907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:16.882060 (+ 153us) server_negotiation.cc:407] Connection header received
0807 08:45:16.882123 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:16.882124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:16.882150 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:16.882238 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:45:16.882482 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.883289 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.884490 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:16.884823 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:16.884888 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:16.885152 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:45:16.885325 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:45:16.885414 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:16.885701 (+ 287us) server_negotiation.cc:299] Negotiation successful
0807 08:45:16.885731 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 08:45:16.895180 10929 raft_consensus.cc:1261] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264 [term 1 FOLLOWER]: Refusing update from remote peer f3ac229a95bf4cafaba94c03fcc6fb24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:45:16.895716 11024 consensus_queue.cc:1035] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521322366bbc4227b01be808d4941264" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:45:16.898128 10780 raft_consensus.cc:1261] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363 [term 1 FOLLOWER]: Refusing update from remote peer f3ac229a95bf4cafaba94c03fcc6fb24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:45:16.898509 11024 consensus_queue.cc:1035] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P f3ac229a95bf4cafaba94c03fcc6fb24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31531a734f7464788ba7fe48d2d8363" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:45:16.900324 11027 mvcc.cc:205] Tried to move back new op lower bound from 7315834129998819328 to 7315834129869115392. Current Snapshot: MvccSnapshot[applied={T|T < 7315834129998819328}]
I20260807 08:45:16.901069 11028 mvcc.cc:205] Tried to move back new op lower bound from 7315834129998819328 to 7315834129869115392. Current Snapshot: MvccSnapshot[applied={T|T < 7315834129998819328}]
I20260807 08:45:16.903998 11031 mvcc.cc:205] Tried to move back new op lower bound from 7315834129998819328 to 7315834129869115392. Current Snapshot: MvccSnapshot[applied={T|T < 7315834129998819328 or (T in {7315834129998819328})}]
I20260807 08:45:16.907231 10475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:45:16.907349 10475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:45:16.908535 10475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88e84ba9332b470db087de11711b0ca8: Sending DeleteTablet for 3 replicas of tablet 84cd134fdddc4d5cb5d4dd2b12f89e67
I20260807 08:45:16.908986 10906 tablet_service.cc:1514] Processing DeleteTablet for tablet 84cd134fdddc4d5cb5d4dd2b12f89e67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:16 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:40092
I20260807 08:45:16.909118 10760 tablet_service.cc:1514] Processing DeleteTablet for tablet 84cd134fdddc4d5cb5d4dd2b12f89e67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:16 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:59490
I20260807 08:45:16.909428 10614 tablet_service.cc:1514] Processing DeleteTablet for tablet 84cd134fdddc4d5cb5d4dd2b12f89e67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:16 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:36322
I20260807 08:45:16.910481 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 10515
I20260807 08:45:16.916391 11040 tablet_replica.cc:330] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363: stopping tablet replica
I20260807 08:45:16.916592 11040 raft_consensus.cc:2229] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:45:16.916745 11040 raft_consensus.cc:2258] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:45:16.921931 11039 tablet_replica.cc:330] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264: stopping tablet replica
I20260807 08:45:16.922255 11039 raft_consensus.cc:2229] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:45:16.922549 11039 raft_consensus.cc:2258] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 08:45:16.923094 10458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:45845 (error 108)
I20260807 08:45:16.923343 11039 ts_tablet_manager.cc:1905] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:45:16.923720 11040 ts_tablet_manager.cc:1905] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P b31531a734f7464788ba7fe48d2d8363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:45:16.923935 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 10692
I20260807 08:45:16.925904 11039 ts_tablet_manager.cc:1918] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:45:16.926462 11039 log.cc:1196] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/84cd134fdddc4d5cb5d4dd2b12f89e67
I20260807 08:45:16.926926 11039 ts_tablet_manager.cc:1939] T 84cd134fdddc4d5cb5d4dd2b12f89e67 P 521322366bbc4227b01be808d4941264: Deleting consensus metadata
I20260807 08:45:16.927634 10461 catalog_manager.cc:4841] TS 521322366bbc4227b01be808d4941264 (127.28.108.131:36431): tablet 84cd134fdddc4d5cb5d4dd2b12f89e67 (table test-table [id=0ff26ce939ab4924870d653894b44d4a]) successfully deleted
I20260807 08:45:16.936827 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 10837
I20260807 08:45:16.942873 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.942594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:45845 (local address 127.0.0.1:36340)
0807 08:45:16.942691 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:16.942790 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 08:45:16.943078 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:16.942691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:42723 (local address 127.0.0.1:59498)
0807 08:45:16.943009 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:16.943044 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.130:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301}
W20260807 08:45:16.943274 10458 catalog_manager.cc:4584] TS f3ac229a95bf4cafaba94c03fcc6fb24 (127.28.108.129:45845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84cd134fdddc4d5cb5d4dd2b12f89e67: Network error: Client connection negotiation failed: client connection to 127.28.108.129:45845: connect: Connection refused (error 111)
I20260807 08:45:16.955490 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 10438
I20260807 08:45:16.966127 29106 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 08:45:17.312203 29106 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 08:45:17.312330 29106 mini_ranger.cc:66] Stopping Ranger...
I20260807 08:45:17.668859 29106 mini_ranger.cc:68] Stopped Ranger
2026-08-07 08:45:17.669 UTC [9413] LOG: received smart shutdown request
2026-08-07 08:45:17.675 UTC [9413] LOG: background worker "logical replication launcher" (PID 9422) exited with exit code 1
2026-08-07 08:45:17.677 UTC [9417] LOG: shutting down
2026-08-07 08:45:17.944 UTC [9413] LOG: database system is shut down
2026-08-07T08:45:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (60013 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:45:18 dist-test-slave-9zdj krb5kdc[11052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:45:18 dist-test-slave-9zdj krb5kdc[11052](info): set up 2 sockets
Aug 07 08:45:18 dist-test-slave-9zdj krb5kdc[11052](info): commencing operation
krb5kdc: starting...
W20260807 08:45:19.278168 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:45:19 dist-test-slave-9zdj krb5kdc[11052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:45:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:45:19.430357 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:41985
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:41985,127.28.108.189:38885,127.28.108.188:39797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:19.525068 11067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:19.525403 11067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:19.525449 11067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:19.526747 11067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:45:19.526852 11067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:19.526876 11067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:19.526901 11067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:45:19.526929 11067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:45:19.528666 11067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:41985,127.28.108.189:38885,127.28.108.188:39797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:41985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:19.529017 11067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:19.529299 11067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:19.535246 11075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:19.535352 11074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:19.535379 11077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:19.536018 11067 server_base.cc:1034] running on GCE node
I20260807 08:45:19.536422 11067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:19.536768 11067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:19.538139 11067 hybrid_clock.cc:648] HybridClock initialized: now 1786092319537942 us; error 76 us; skew 500 ppm
W20260807 08:45:20.526027 11067 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-9zdj': real 0.987s user 0.001s sys 0.000s
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:20.528790 11067 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:45:20.529605 11067 webserver.cc:460] Webserver started at http://127.28.108.190:37733/ using document root <none> and password file <none>
I20260807 08:45:20.529974 11067 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:20.530062 11067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:20.530260 11067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:20.531584 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09"
format_stamp: "Formatted at 2026-08-07 08:45:20 on dist-test-slave-9zdj"
server_key: "cd29f1668c37c432350ce4120591bba4"
server_key_iv: "728d08002531e6851491d6e9d3c8c13c"
server_key_version: "encryptionkey@0"
I20260807 08:45:20.532042 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09"
format_stamp: "Formatted at 2026-08-07 08:45:20 on dist-test-slave-9zdj"
server_key: "cd29f1668c37c432350ce4120591bba4"
server_key_iv: "728d08002531e6851491d6e9d3c8c13c"
server_key_version: "encryptionkey@0"
I20260807 08:45:20.533836 11067 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 08:45:20.534957 11085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:20.535488 11067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:45:20.535605 11067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09"
format_stamp: "Formatted at 2026-08-07 08:45:20 on dist-test-slave-9zdj"
server_key: "cd29f1668c37c432350ce4120591bba4"
server_key_iv: "728d08002531e6851491d6e9d3c8c13c"
server_key_version: "encryptionkey@0"
I20260807 08:45:20.535739 11067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:20.561909 11067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:20.565049 11067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:20.565222 11067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:20.570362 11067 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:41985
I20260807 08:45:20.570494 11141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:41985 every 8 connection(s)
I20260807 08:45:20.570952 11067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:45:20.572146 11142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:20.574429 11142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:45:20.581141 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 11067
I20260807 08:45:20.581394 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:45:20.581703 29106 external_mini_cluster.cc:1428] Setting key e703db4ca61dee181f26ce382fbb918e
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:20.589080 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.575600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.28.108.190:41985)
0807 08:45:20.575769 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:20.575773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:20.575805 (+ 32us) server_negotiation.cc:407] Connection header received
0807 08:45:20.576349 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:20.576352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:20.576373 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:20.576461 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:20.577185 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.578388 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.579517 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.579896 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.579994 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.583010 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:20.583021 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:20.583021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:20.583039 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:20.586815 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.587535 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.587539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.587540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.587661 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.588034 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.588035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.588035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.588322 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:20.588409 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:20.588713 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 08:45:20.588874 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260807 08:45:20.589238 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.575304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41985 (local address 127.0.0.1:46508)
0807 08:45:20.575555 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.575614 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:20.576158 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:20.576501 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:20.576503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:20.576796 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:20.576989 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.576993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.578511 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.578512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.579408 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.579412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.580010 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.580011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.580093 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.580956 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:20.580968 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:20.582877 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:20.586981 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.586996 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.586999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.587394 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.587762 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.587763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.587763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.587936 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.588412 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:20.588422 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:20.588536 (+ 114us) client_negotiation.cc:769] Sending connection context
0807 08:45:20.588676 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 08:45:20.589001 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":96,"thread_start_us":91,"threads_started":1}
I20260807 08:45:20.590548 11142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:45:20.591053 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.590865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:38885 (local address 127.0.0.1:54468)
0807 08:45:20.590936 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.591008 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260807 08:45:20.591255 11088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.108.189:38885: connect: Connection refused (error 111)
I20260807 08:45:20.591796 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.591661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:38885 (local address 127.0.0.1:54474)
0807 08:45:20.591722 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.591759 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260807 08:45:20.591966 11142 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.189:38885: Network error: Client connection negotiation failed: client connection to 127.28.108.189:38885: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.28.108.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab.
Entry for principal kudu/127.28.108.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab.
I20260807 08:45:20.618399 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.189:38885
--webserver_interface=127.28.108.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab
--principal=kudu/127.28.108.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:41985,127.28.108.189:38885,127.28.108.188:39797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:45:20.641225 11142 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } attempt: 1
I20260807 08:45:20.641944 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.641747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:38885 (local address 127.0.0.1:54480)
0807 08:45:20.641825 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.641908 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 08:45:20.642349 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.642238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:38885 (local address 127.0.0.1:54494)
0807 08:45:20.642290 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.642320 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260807 08:45:20.642454 11142 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.189:38885: Network error: Client connection negotiation failed: client connection to 127.28.108.189:38885: connect: Connection refused (error 111)
I20260807 08:45:20.710619 11142 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } attempt: 2
I20260807 08:45:20.711252 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.711045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:38885 (local address 127.0.0.1:54506)
0807 08:45:20.711132 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.711209 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 08:45:20.711706 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.711589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:38885 (local address 127.0.0.1:54512)
0807 08:45:20.711645 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.711680 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260807 08:45:20.711841 11142 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.189:38885: Network error: Client connection negotiation failed: client connection to 127.28.108.189:38885: connect: Connection refused (error 111)
W20260807 08:45:20.711750 11150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:20.712066 11150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:20.712098 11150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:20.713445 11150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:45:20.713505 11150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:20.713519 11150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:20.713533 11150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:45:20.713547 11150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:45:20.715426 11150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:41985,127.28.108.189:38885,127.28.108.188:39797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab
--principal=kudu/127.28.108.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.189:38885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.108.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:20.715749 11150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:20.716054 11150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:20.722332 11159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:20.722332 11157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:20.722501 11150 server_base.cc:1034] running on GCE node
W20260807 08:45:20.722332 11156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:20.723012 11150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:20.723397 11150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:20.724651 11150 hybrid_clock.cc:648] HybridClock initialized: now 1786092320724622 us; error 40 us; skew 500 ppm
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:20.728168 11150 init.cc:377] Logged in from keytab as kudu/127.28.108.189@KRBTEST.COM (short username kudu)
I20260807 08:45:20.728739 11150 webserver.cc:460] Webserver started at http://127.28.108.189:46207/ using document root <none> and password file <none>
I20260807 08:45:20.729031 11150 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:20.729110 11150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:20.729285 11150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:20.730611 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/data/instance:
uuid: "c6514f3b44c04062b56ec4131e5c62b9"
format_stamp: "Formatted at 2026-08-07 08:45:20 on dist-test-slave-9zdj"
server_key: "0c4c2af9f9a1489bede59facd7436829"
server_key_iv: "6a01c0abb13177c1dd512bdc1ffcf17f"
server_key_version: "encryptionkey@0"
I20260807 08:45:20.731024 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/wal/instance:
uuid: "c6514f3b44c04062b56ec4131e5c62b9"
format_stamp: "Formatted at 2026-08-07 08:45:20 on dist-test-slave-9zdj"
server_key: "0c4c2af9f9a1489bede59facd7436829"
server_key_iv: "6a01c0abb13177c1dd512bdc1ffcf17f"
server_key_version: "encryptionkey@0"
I20260807 08:45:20.732754 11150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 08:45:20.733986 11166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:20.734386 11150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:20.734543 11150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/wal
uuid: "c6514f3b44c04062b56ec4131e5c62b9"
format_stamp: "Formatted at 2026-08-07 08:45:20 on dist-test-slave-9zdj"
server_key: "0c4c2af9f9a1489bede59facd7436829"
server_key_iv: "6a01c0abb13177c1dd512bdc1ffcf17f"
server_key_version: "encryptionkey@0"
I20260807 08:45:20.734709 11150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:20.753077 11150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:20.756192 11150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:20.756421 11150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:20.761705 11150 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.189:38885
I20260807 08:45:20.761758 11223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.189:38885 every 8 connection(s)
I20260807 08:45:20.762226 11150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
I20260807 08:45:20.763386 11224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:20.765408 11224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:45:20.766623 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 11150
I20260807 08:45:20.766793 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-1/wal/instance
I20260807 08:45:20.767043 29106 external_mini_cluster.cc:1428] Setting key 266600d3d38b62b1c7cfb586fd694203
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.108.188@KRBTEST.COM; defaulting to no policy
I20260807 08:45:20.779641 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.766473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.28.108.190:41985)
0807 08:45:20.766581 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:20.766583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:20.766984 (+ 401us) server_negotiation.cc:407] Connection header received
0807 08:45:20.767891 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:20.767893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:20.767909 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:20.767973 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:20.769118 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.769998 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.772164 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.772390 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.772464 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.775989 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:20.775991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:20.775991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:20.776000 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:20.777817 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.778404 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.778406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.778407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.778484 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.778816 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.778817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.778817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.778964 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:20.779041 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:20.779417 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 08:45:20.779540 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 08:45:20.779892 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.766418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41985 (local address 127.0.0.1:46520)
0807 08:45:20.766688 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.766766 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:20.767703 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:20.768089 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:20.768094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:20.768591 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:20.768961 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.768968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.770144 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.770145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.772014 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.772020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.772523 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.772524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.772741 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.773550 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:20.773564 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:20.775822 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:20.777949 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.777965 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.777968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.778300 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.778567 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.778568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.778569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.778730 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.779045 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:20.779056 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:20.779245 (+ 189us) client_negotiation.cc:769] Sending connection context
0807 08:45:20.779379 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 08:45:20.779587 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":133,"threads_started":1}
I20260807 08:45:20.781239 11224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.188@KRBTEST.COM" created.
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
I20260807 08:45:20.792398 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.781677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:38885 (local address 127.0.0.1:54522)
0807 08:45:20.781788 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.781804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:20.781864 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:20.782558 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:20.782560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:20.782752 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:20.782823 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.782825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.784295 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.784296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.785221 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.785226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.785987 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.785988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.786076 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.786734 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:20.786739 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:20.788222 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:20.790961 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.790962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.790963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.791167 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.791447 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.791447 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.791448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.791516 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.791882 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:20.791883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:20.791927 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 08:45:20.791999 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 08:45:20.792165 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260807 08:45:20.792673 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.781855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.28.108.189:38885)
0807 08:45:20.782284 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:20.782290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:20.782357 (+ 67us) server_negotiation.cc:407] Connection header received
0807 08:45:20.782383 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:20.782385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:20.782399 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:20.782487 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:20.782980 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.784154 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.785436 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.785883 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.785990 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.788364 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:20.788386 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:20.788386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:20.788394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:20.790799 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.791295 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.791300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.791301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.791375 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.791593 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.791594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.791594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.791798 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:20.791896 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:20.792043 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 08:45:20.792379 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":389,"thread_start_us":353,"threads_started":1}
I20260807 08:45:20.793709 11224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:45:20.794207 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.794047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53250)
0807 08:45:20.794112 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.794179 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 08:45:20.794426 11171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
I20260807 08:45:20.795009 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.794865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53266)
0807 08:45:20.794941 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.794980 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 08:45:20.795230 11224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:39797: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
I20260807 08:45:20.803052 11142 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab.
Entry for principal kudu/127.28.108.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab.
I20260807 08:45:20.806550 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.188:39797
--webserver_interface=127.28.108.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab
--principal=kudu/127.28.108.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:41985,127.28.108.189:38885,127.28.108.188:39797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
I20260807 08:45:20.814116 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.803570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:38885 (local address 127.0.0.1:54532)
0807 08:45:20.803643 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.803657 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:20.803717 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:20.804009 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:20.804010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:20.804200 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:20.804277 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.804279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.805278 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.805280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.806201 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.806205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.806928 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.806930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.807031 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.807727 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:20.807735 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:20.809693 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:20.812510 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.812512 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.812513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.812754 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.813062 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.813063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.813064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.813143 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.813778 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:20.813779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:20.813862 (+ 83us) client_negotiation.cc:769] Sending connection context
0807 08:45:20.813927 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 08:45:20.814037 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 08:45:20.814245 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.803741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.28.108.189:38885)
0807 08:45:20.803795 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:20.803797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:20.803813 (+ 16us) server_negotiation.cc:407] Connection header received
0807 08:45:20.803851 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:20.803852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:20.803869 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:20.803954 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:20.804388 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.805070 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.806325 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.806758 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.806850 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.809874 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:20.809876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:20.809877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:20.809888 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:20.812312 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.812854 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.812857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.812857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.812942 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.813347 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.813349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.813349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.813543 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:20.813651 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:20.813980 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 08:45:20.814100 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 08:45:20.814885 11142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:45:20.815315 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.815166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53274)
0807 08:45:20.815228 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.815275 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260807 08:45:20.815750 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.815630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53278)
0807 08:45:20.815684 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.815717 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260807 08:45:20.815877 11142 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:39797: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
I20260807 08:45:20.844564 11224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } attempt: 1
I20260807 08:45:20.845270 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.845090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53280)
0807 08:45:20.845187 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.845234 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 08:45:20.845899 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.845745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53292)
0807 08:45:20.845828 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.845870 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 08:45:20.846040 11224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:39797: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
I20260807 08:45:20.847044 11142 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } attempt: 1
I20260807 08:45:20.847705 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.847550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53296)
0807 08:45:20.847619 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.847667 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260807 08:45:20.848356 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.848226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53300)
0807 08:45:20.848289 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.848328 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 08:45:20.848483 11142 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:39797: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
W20260807 08:45:20.905171 11232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:20.905473 11232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:20.905505 11232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:20.907127 11232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:45:20.907541 11232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:20.907563 11232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:20.907579 11232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:45:20.907596 11232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:45:20.909711 11232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:41985,127.28.108.189:38885,127.28.108.188:39797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab
--principal=kudu/127.28.108.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.188:39797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.108.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:20.909982 11232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:20.910212 11232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:45:20.914254 11224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } attempt: 2
I20260807 08:45:20.914812 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.914660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53308)
0807 08:45:20.914716 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.914759 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260807 08:45:20.915393 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.915239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53310)
0807 08:45:20.915324 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.915366 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260807 08:45:20.915609 11224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:39797: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
W20260807 08:45:20.916335 11238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:20.916467 11241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:20.916508 11232 server_base.cc:1034] running on GCE node
W20260807 08:45:20.916467 11239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:20.917434 11232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:20.917943 11232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:20.919163 11232 hybrid_clock.cc:648] HybridClock initialized: now 1786092320919152 us; error 52 us; skew 500 ppm
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:20.923049 11232 init.cc:377] Logged in from keytab as kudu/127.28.108.188@KRBTEST.COM (short username kudu)
I20260807 08:45:20.923729 11232 webserver.cc:460] Webserver started at http://127.28.108.188:43381/ using document root <none> and password file <none>
I20260807 08:45:20.923662 11142 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } attempt: 2
I20260807 08:45:20.924293 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.924141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53324)
0807 08:45:20.924215 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.924260 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 08:45:20.924321 11232 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:20.924446 11232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:20.924672 11232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:20.924865 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.924639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53338)
0807 08:45:20.924784 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.924831 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 08:45:20.924965 11142 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:39797: Network error: Client connection negotiation failed: client connection to 127.28.108.188:39797: connect: Connection refused (error 111)
I20260807 08:45:20.926415 11232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/data/instance:
uuid: "445e5c8178b544d3943b68726598117d"
format_stamp: "Formatted at 2026-08-07 08:45:20 on dist-test-slave-9zdj"
server_key: "f4b35658ac468a7a078cf11f0cae6413"
server_key_iv: "9586ad523d85ee2193513e0ed15cabf3"
server_key_version: "encryptionkey@0"
I20260807 08:45:20.926929 11232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/wal/instance:
uuid: "445e5c8178b544d3943b68726598117d"
format_stamp: "Formatted at 2026-08-07 08:45:20 on dist-test-slave-9zdj"
server_key: "f4b35658ac468a7a078cf11f0cae6413"
server_key_iv: "9586ad523d85ee2193513e0ed15cabf3"
server_key_version: "encryptionkey@0"
I20260807 08:45:20.928726 11232 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 08:45:20.929970 11250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:20.930387 11232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:20.930546 11232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/wal
uuid: "445e5c8178b544d3943b68726598117d"
format_stamp: "Formatted at 2026-08-07 08:45:20 on dist-test-slave-9zdj"
server_key: "f4b35658ac468a7a078cf11f0cae6413"
server_key_iv: "9586ad523d85ee2193513e0ed15cabf3"
server_key_version: "encryptionkey@0"
I20260807 08:45:20.930663 11232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:20.954932 11232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:20.958177 11232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:20.958459 11232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:20.963394 11232 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.188:39797
I20260807 08:45:20.963481 11306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.188:39797 every 8 connection(s)
I20260807 08:45:20.963904 11232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
I20260807 08:45:20.965099 11307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:20.965871 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 11232
I20260807 08:45:20.966039 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786092133523677-29106-0/minicluster-data/master-2/wal/instance
I20260807 08:45:20.966358 29106 external_mini_cluster.cc:1428] Setting key de997c72866ca0502da6db3526844e39
I20260807 08:45:20.967584 11307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
I20260807 08:45:20.980487 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.968880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.28.108.190:41985)
0807 08:45:20.968942 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:20.968945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:20.969027 (+ 82us) server_negotiation.cc:407] Connection header received
0807 08:45:20.969075 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:20.969076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:20.969088 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:20.969166 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:20.969855 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.970626 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.972083 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.972406 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.972501 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.975840 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:20.975842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:20.975843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:20.975855 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:20.978355 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.978977 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.978979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.978979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.979066 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.979450 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.979452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.979452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.979648 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:20.979709 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:20.980134 (+ 425us) server_negotiation.cc:299] Negotiation successful
0807 08:45:20.980367 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 08:45:20.980736 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.968724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41985 (local address 127.0.0.1:46532)
0807 08:45:20.968904 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.968946 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:20.969016 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:20.969217 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:20.969220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:20.969538 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:20.969736 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.969742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.970753 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.970754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.971980 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.971983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.972523 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.972524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.972612 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.973361 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:20.973373 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:20.975640 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:20.978469 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.978478 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.978481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.978886 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.979153 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.979154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.979154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.979336 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.979732 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:20.979741 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:20.979958 (+ 217us) client_negotiation.cc:769] Sending connection context
0807 08:45:20.980055 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 08:45:20.980425 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260807 08:45:20.980738 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.967057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.28.108.188:39797)
0807 08:45:20.967258 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:20.967269 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:20.967312 (+ 43us) server_negotiation.cc:407] Connection header received
0807 08:45:20.967811 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:20.967819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:20.967874 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:20.968002 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:20.968967 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.970433 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.971725 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.972245 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.972383 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.976575 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:20.976585 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:20.976585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:20.976594 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:20.978909 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.979460 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.979464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.979465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.979527 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.979846 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.979847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.979848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.980004 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:20.980207 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:20.980311 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 08:45:20.980542 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260807 08:45:20.981492 11307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:20.991211 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.981951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.28.108.189:38885)
0807 08:45:20.982020 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:20.982024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:20.982158 (+ 134us) server_negotiation.cc:407] Connection header received
0807 08:45:20.982308 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:20.982310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:20.982324 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:20.982394 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:20.982695 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.983418 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.984336 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.984732 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.984822 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.987510 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:20.987511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:20.987512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:20.987520 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:20.989470 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.990027 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.990028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.990028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.990092 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.990351 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.990351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.990352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.990519 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:20.990604 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:20.991001 (+ 397us) server_negotiation.cc:299] Negotiation successful
0807 08:45:20.991116 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 08:45:20.991139 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.981855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:38885 (local address 127.0.0.1:54536)
0807 08:45:20.982086 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.982097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:20.982152 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:20.982397 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:20.982398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:20.982554 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:20.982597 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.982598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.983518 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.983519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.984238 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.984241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.984858 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.984859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.984929 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.985593 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:20.985598 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:20.987371 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:20.989591 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.989594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.989594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.989931 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.990177 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:20.990178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:20.990179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:20.990260 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:20.990624 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:20.990625 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:20.990774 (+ 149us) client_negotiation.cc:769] Sending connection context
0807 08:45:20.990887 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 08:45:20.991015 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260807 08:45:20.991794 11307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:45:20.992069 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.982091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.28.108.190:41985)
0807 08:45:20.982131 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:20.982133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:20.982238 (+ 105us) server_negotiation.cc:407] Connection header received
0807 08:45:20.982310 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:20.982311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:20.982321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:20.982374 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:20.983128 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.983882 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.984866 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.985156 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.985328 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.988193 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:20.988195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:20.988195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:20.988204 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:20.990113 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.990748 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.990750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.990750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.990832 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:20.991132 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:20.991133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:20.991134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:20.991306 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:20.991567 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:20.991831 (+ 264us) server_negotiation.cc:299] Negotiation successful
0807 08:45:20.991933 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
Aug 07 08:45:20 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
I20260807 08:45:21.001725 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.992157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53348)
0807 08:45:20.992342 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:20.992355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:20.992405 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:20.992568 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:20.992569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:20.992769 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:20.992828 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.992829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.993890 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.993891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.994709 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:20.994712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.995341 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.995343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:20.995396 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.996031 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:20.996037 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:20.997656 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:21.000025 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:21.000026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:21.000026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:21.000391 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:21.000780 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:21.000781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:21.000782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:21.000854 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:21.001213 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:21.001214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:21.001382 (+ 168us) client_negotiation.cc:769] Sending connection context
0807 08:45:21.001471 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 08:45:21.001593 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260807 08:45:21.001852 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.992337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.28.108.188:39797)
0807 08:45:20.992413 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:20.992415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:20.992427 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:45:20.992453 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:20.992454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:20.992466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:20.992563 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:20.992929 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.993803 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.994806 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.995117 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.995210 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.997764 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:20.997765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:20.997766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:20.997773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:20.999885 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.000530 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.000532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.000532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.000605 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.000935 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.000937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.000937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.001112 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:21.001226 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:21.001556 (+ 330us) server_negotiation.cc:299] Negotiation successful
0807 08:45:21.001731 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 08:45:21.002703 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:20.993010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.28.108.189:38885)
0807 08:45:20.993052 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:20.993053 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:20.993062 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:45:20.993116 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:20.993117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:20.993125 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:20.993170 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:20.993891 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.994621 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.995644 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:20.995900 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:20.995999 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:20.999121 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:20.999140 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:20.999140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:20.999150 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:21.001182 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.001683 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.001685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.001686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.001810 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.002102 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.002103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.002104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.002284 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:21.002528 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:21.002565 (+ 37us) server_negotiation.cc:299] Negotiation successful
0807 08:45:21.002637 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 08:45:21.003861 11307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d: Bootstrap starting.
I20260807 08:45:21.004952 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:45:21.005007 11307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:21.005373 11307 log.cc:824] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:21.006721 11307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d: No bootstrap required, opened a new log
I20260807 08:45:21.006855 11224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } attempt: 3
I20260807 08:45:21.009105 11307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } }
I20260807 08:45:21.009330 11307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:21.009377 11307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445e5c8178b544d3943b68726598117d, State: Initialized, Role: FOLLOWER
I20260807 08:45:21.009687 11307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } }
I20260807 08:45:21.010083 11307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:45:21.011210 11321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } } }
I20260807 08:45:21.011339 11321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [sys.catalog]: This master's current role is: FOLLOWER
W20260807 08:45:21.012375 11334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:45:21.012446 11334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 07 08:45:21 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
I20260807 08:45:21.013823 11232 master_runner.cc:186] Error getting master registration for 127.28.108.190:41985: OK, instance_id { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" instance_seqno: 1786092320568815 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 08:45:21.017374 11232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 08:45:21.017843 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:21.007142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53358)
0807 08:45:21.007227 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:21.007238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:21.007283 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:21.007773 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:21.007775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:21.008001 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:21.008072 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:21.008074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.009246 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.009248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:21.010122 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:21.010126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.010754 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.010755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:21.010841 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:21.011734 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:21.011739 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:21.013710 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:21.016436 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:21.016437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:21.016438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:21.016690 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:21.016956 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:21.016957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:21.016957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:21.017021 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:21.017395 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:21.017396 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:21.017448 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 08:45:21.017524 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 08:45:21.017690 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 08:45:21.018396 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:21.007207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.28.108.188:39797)
0807 08:45:21.007587 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:21.007590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:21.007600 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:45:21.007627 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:21.007628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:21.007652 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:21.007741 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:21.008193 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.009139 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.010239 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.010613 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.011068 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:21.013868 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:21.013869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:21.013870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:21.013878 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:21.016331 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.016784 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.016785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.016786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.016882 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.017090 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.017091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.017091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.017294 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:21.017364 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:21.018107 (+ 743us) server_negotiation.cc:299] Negotiation successful
0807 08:45:21.018261 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355}
I20260807 08:45:21.020779 11224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9: Bootstrap starting.
I20260807 08:45:21.021767 11224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:21.022162 11224 log.cc:824] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:21.022218 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:21.007424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.28.108.188:39797)
0807 08:45:21.009556 (+ 2132us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:21.009559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:21.011925 (+ 2366us) server_negotiation.cc:407] Connection header received
0807 08:45:21.012670 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:21.012672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:21.012685 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:21.012720 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:21.013821 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.014739 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.015924 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.016252 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.016303 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:21.018047 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:21.018048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:21.018048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:21.018056 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:21.020463 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.021001 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.021003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.021003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.021076 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.021336 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.021337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.021337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.021495 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:21.021589 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:21.021984 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 08:45:21.022087 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2101,"thread_start_us":99,"threads_started":1}
I20260807 08:45:21.023286 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:21.007037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54546 (local address 127.28.108.189:38885)
0807 08:45:21.007200 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:21.007203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:21.012033 (+ 4830us) server_negotiation.cc:407] Connection header received
0807 08:45:21.012163 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:21.012165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:21.012178 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:21.012212 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:21.014780 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.015617 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.016805 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.017156 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.017201 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:21.019698 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:21.019700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:21.019700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:21.019708 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:21.021712 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.022196 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.022198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.022199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.022264 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.022514 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.022514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.022515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.022667 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:21.022799 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:21.023054 (+ 255us) server_negotiation.cc:299] Negotiation successful
0807 08:45:21.023160 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260807 08:45:21.024099 11142 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } attempt: 3
I20260807 08:45:21.024454 11224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9: No bootstrap required, opened a new log
I20260807 08:45:21.026228 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:21.007032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.28.108.190:41985)
0807 08:45:21.007087 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:21.007090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:21.007603 (+ 513us) server_negotiation.cc:407] Connection header received
0807 08:45:21.007668 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:21.007670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:21.007684 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:21.007752 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:21.008581 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.009387 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.010444 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.010723 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.010806 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:21.013024 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:21.013026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:21.013027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:21.013035 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:21.024295 (+ 11260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.024775 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.024776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.024777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.024857 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.025100 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.025101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.025101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.025295 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:21.025943 (+ 648us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:21.025992 (+ 49us) server_negotiation.cc:299] Negotiation successful
0807 08:45:21.026083 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 08:45:21.027851 11224 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } }
I20260807 08:45:21.028158 11224 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:21.028234 11224 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6514f3b44c04062b56ec4131e5c62b9, State: Initialized, Role: FOLLOWER
I20260807 08:45:21.028394 11224 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } }
I20260807 08:45:21.028802 11337 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } } }
I20260807 08:45:21.028856 11224 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:45:21.028998 11337 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [sys.catalog]: This master's current role is: FOLLOWER
Aug 07 08:45:21 dist-test-slave-9zdj krb5kdc[11052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
I20260807 08:45:21.033545 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:21.024676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.28.108.188:39797)
0807 08:45:21.024737 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:21.024739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:21.024747 (+ 8us) server_negotiation.cc:407] Connection header received
0807 08:45:21.024771 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:21.024772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:21.024783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:21.024835 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:21.025253 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.026107 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.027175 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.027417 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.027511 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:21.030100 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:21.030102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:21.030102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:21.030111 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:21.032023 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.032496 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.032497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.032498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.032575 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:21.032820 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:21.032822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:21.032822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:21.032985 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:21.033061 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:21.033330 (+ 269us) server_negotiation.cc:299] Negotiation successful
0807 08:45:21.033419 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 08:45:21.033537 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:21.024425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:39797 (local address 127.0.0.1:53386)
0807 08:45:21.024532 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:21.024544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:21.024590 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:21.024886 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:21.024887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:21.025077 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:21.025159 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:21.025161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.026206 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.026207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:21.027041 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:21.027045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:21.027508 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:21.027509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:21.027571 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:21.028109 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:21.028115 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:21.029974 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:21.032128 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:21.032129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:21.032130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:21.032398 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:21.032659 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:21.032660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:21.032661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:21.032736 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:21.033098 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:21.033099 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:21.033157 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 08:45:21.033279 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 08:45:21.033415 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 08:45:21.031498 11150 master_runner.cc:186] Error getting master registration for 127.28.108.190:41985: OK, instance_id { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" instance_seqno: 1786092320568815 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 08:45:21.034756 11150 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 08:45:21.035630 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09: Bootstrap starting.
I20260807 08:45:21.036511 11142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:21.036859 11142 log.cc:824] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:21.037978 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09: No bootstrap required, opened a new log
I20260807 08:45:21.039855 11142 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } }
I20260807 08:45:21.040076 11142 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:21.040140 11142 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9dbbbd6a9e434b94f1397b1b3b1c09, State: Initialized, Role: FOLLOWER
I20260807 08:45:21.040297 11142 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } }
I20260807 08:45:21.040663 11353 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } } }
I20260807 08:45:21.040743 11142 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:45:21.040776 11353 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 08:45:21.042415 11350 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:45:21.042484 11350 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260807 08:45:21.043277 11367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:45:21.043398 11367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:45:21.046051 11067 master_runner.cc:263] No leader master found from master ff9dbbbd6a9e434b94f1397b1b3b1c09
I20260807 08:45:21.046133 11067 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260807 08:45:21.066267 11092 tablet.cc:2369] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:45:21.166002 11337 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:45:21.166180 11337 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } }
I20260807 08:45:21.166464 11337 leader_election.cc:288] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff9dbbbd6a9e434b94f1397b1b3b1c09 (127.28.108.190:41985), 445e5c8178b544d3943b68726598117d (127.28.108.188:39797)
I20260807 08:45:21.166877 11281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6514f3b44c04062b56ec4131e5c62b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445e5c8178b544d3943b68726598117d" is_pre_election: true
I20260807 08:45:21.167076 11281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6514f3b44c04062b56ec4131e5c62b9 in term 0.
I20260807 08:45:21.167425 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6514f3b44c04062b56ec4131e5c62b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" is_pre_election: true
I20260807 08:45:21.167583 11114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6514f3b44c04062b56ec4131e5c62b9 in term 0.
I20260807 08:45:21.167862 11171 leader_election.cc:302] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6514f3b44c04062b56ec4131e5c62b9, ff9dbbbd6a9e434b94f1397b1b3b1c09; no voters:
I20260807 08:45:21.168087 11337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:45:21.168157 11337 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:45:21.168191 11337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:21.169251 11337 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } }
I20260807 08:45:21.169414 11337 leader_election.cc:288] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [CANDIDATE]: Term 1 election: Requested vote from peers ff9dbbbd6a9e434b94f1397b1b3b1c09 (127.28.108.190:41985), 445e5c8178b544d3943b68726598117d (127.28.108.188:39797)
I20260807 08:45:21.169785 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6514f3b44c04062b56ec4131e5c62b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09"
I20260807 08:45:21.169886 11114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:21.170953 11114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6514f3b44c04062b56ec4131e5c62b9 in term 1.
I20260807 08:45:21.171168 11281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6514f3b44c04062b56ec4131e5c62b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445e5c8178b544d3943b68726598117d"
I20260807 08:45:21.171276 11281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:21.172438 11281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6514f3b44c04062b56ec4131e5c62b9 in term 1.
I20260807 08:45:21.172674 11171 leader_election.cc:302] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445e5c8178b544d3943b68726598117d, c6514f3b44c04062b56ec4131e5c62b9; no voters:
I20260807 08:45:21.172811 11337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:45:21.172978 11337 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [term 1 LEADER]: Becoming Leader. State: Replica: c6514f3b44c04062b56ec4131e5c62b9, State: Running, Role: LEADER
I20260807 08:45:21.173094 11337 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } }
I20260807 08:45:21.173556 11337 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6514f3b44c04062b56ec4131e5c62b9. Latest consensus state: current_term: 1 leader_uuid: "c6514f3b44c04062b56ec4131e5c62b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } } }
I20260807 08:45:21.173694 11337 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:21.174546 11372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:45:21.174768 11372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:45:21.176620 11281 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [term 1 FOLLOWER]: Refusing update from remote peer c6514f3b44c04062b56ec4131e5c62b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:45:21.177034 11370 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:45:21.179544 11321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6514f3b44c04062b56ec4131e5c62b9. Latest consensus state: current_term: 1 leader_uuid: "c6514f3b44c04062b56ec4131e5c62b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } } }
I20260807 08:45:21.179680 11321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:45:21.180197 11114 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [term 1 FOLLOWER]: Refusing update from remote peer c6514f3b44c04062b56ec4131e5c62b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:45:21.180539 11370 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:45:21.182374 11353 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6514f3b44c04062b56ec4131e5c62b9. Latest consensus state: current_term: 1 leader_uuid: "c6514f3b44c04062b56ec4131e5c62b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } } }
I20260807 08:45:21.182539 11353 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:45:21.185230 11370 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6514f3b44c04062b56ec4131e5c62b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } } }
I20260807 08:45:21.185377 11370 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:21.186015 11321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6514f3b44c04062b56ec4131e5c62b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } } }
I20260807 08:45:21.186127 11321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:45:21.186378 11373 mvcc.cc:205] Tried to move back new op lower bound from 7315834147536228352 to 7315834147525713920. Current Snapshot: MvccSnapshot[applied={T|T < 7315834147536228352}]
I20260807 08:45:21.186503 11374 mvcc.cc:205] Tried to move back new op lower bound from 7315834147536228352 to 7315834147525713920. Current Snapshot: MvccSnapshot[applied={T|T < 7315834147536228352}]
I20260807 08:45:21.187531 11337 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6514f3b44c04062b56ec4131e5c62b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } } }
I20260807 08:45:21.187639 11337 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:21.188195 11353 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6514f3b44c04062b56ec4131e5c62b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9dbbbd6a9e434b94f1397b1b3b1c09" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41985 } } peers { permanent_uuid: "c6514f3b44c04062b56ec4131e5c62b9" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 38885 } } peers { permanent_uuid: "445e5c8178b544d3943b68726598117d" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 39797 } } }
I20260807 08:45:21.188314 11353 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:45:21.188541 11376 mvcc.cc:205] Tried to move back new op lower bound from 7315834147536228352 to 7315834147525713920. Current Snapshot: MvccSnapshot[applied={T|T < 7315834147536228352}]
I20260807 08:45:21.190099 11372 catalog_manager.cc:1347] Generated new cluster ID: da7e48cbf7d64da1b560c5c274d76a90
I20260807 08:45:21.190172 11372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:45:21.225494 11372 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:45:21.226389 11372 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:45:21.236258 11372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6514f3b44c04062b56ec4131e5c62b9: Generated new TSK 0
I20260807 08:45:21.236554 11372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:45:22.013031 11334 catalog_manager.cc:1259] Loaded cluster ID: da7e48cbf7d64da1b560c5c274d76a90
I20260807 08:45:22.013126 11334 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d: loading cluster ID for follower catalog manager: success
I20260807 08:45:22.014034 11334 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d: acquiring CA information for follower catalog manager: success
I20260807 08:45:22.014328 11334 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 445e5c8178b544d3943b68726598117d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 08:45:22.044008 11367 catalog_manager.cc:1259] Loaded cluster ID: da7e48cbf7d64da1b560c5c274d76a90
I20260807 08:45:22.044103 11367 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09: loading cluster ID for follower catalog manager: success
I20260807 08:45:22.045222 11367 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09: acquiring CA information for follower catalog manager: success
I20260807 08:45:22.045670 11367 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ff9dbbbd6a9e434b94f1397b1b3b1c09: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 08:45:22.299892 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 11067
I20260807 08:45:22.309484 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 11150
I20260807 08:45:22.318965 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 11232
2026-08-07T08:45:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4241 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:45:22 dist-test-slave-9zdj krb5kdc[11389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:45:22 dist-test-slave-9zdj krb5kdc[11389](info): set up 2 sockets
Aug 07 08:45:22 dist-test-slave-9zdj krb5kdc[11389](info): commencing operation
krb5kdc: starting...
W20260807 08:45:23.503057 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:45:23 dist-test-slave-9zdj krb5kdc[11389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:45:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:45:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:45:23.663161 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42301
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:42301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:23.766244 11402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:23.766510 11402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:23.766538 11402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:23.768013 11402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:45:23.768085 11402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:23.768102 11402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:23.768117 11402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:45:23.768131 11402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:45:23.770177 11402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:42301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:23.770550 11402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:23.770856 11402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:23.777166 11408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:23.777334 11409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:23.777280 11402 server_base.cc:1034] running on GCE node
W20260807 08:45:23.777388 11411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:23.777830 11402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:23.778267 11402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:23.779543 11402 hybrid_clock.cc:648] HybridClock initialized: now 1786092323779518 us; error 43 us; skew 500 ppm
Aug 07 08:45:23 dist-test-slave-9zdj krb5kdc[11389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092323, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:23.783290 11402 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:45:23.784240 11402 webserver.cc:460] Webserver started at http://127.28.108.190:34513/ using document root <none> and password file <none>
I20260807 08:45:23.784566 11402 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:23.784657 11402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:23.784868 11402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:23.786563 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "cb5df6d7850d4112b0ab28624d02be9d"
format_stamp: "Formatted at 2026-08-07 08:45:23 on dist-test-slave-9zdj"
server_key: "a49b530c21d39d7f36cbb11b3ecfdd52"
server_key_iv: "7e27dec759834a6f4915a2c71e223c4a"
server_key_version: "encryptionkey@0"
I20260807 08:45:23.787076 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "cb5df6d7850d4112b0ab28624d02be9d"
format_stamp: "Formatted at 2026-08-07 08:45:23 on dist-test-slave-9zdj"
server_key: "a49b530c21d39d7f36cbb11b3ecfdd52"
server_key_iv: "7e27dec759834a6f4915a2c71e223c4a"
server_key_version: "encryptionkey@0"
I20260807 08:45:23.788904 11402 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 08:45:23.790369 11420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:23.790733 11402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:45:23.790943 11402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "cb5df6d7850d4112b0ab28624d02be9d"
format_stamp: "Formatted at 2026-08-07 08:45:23 on dist-test-slave-9zdj"
server_key: "a49b530c21d39d7f36cbb11b3ecfdd52"
server_key_iv: "7e27dec759834a6f4915a2c71e223c4a"
server_key_version: "encryptionkey@0"
I20260807 08:45:23.791081 11402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:23.804998 11402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:23.808228 11402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:23.808451 11402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:23.813524 11402 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:42301
I20260807 08:45:23.813591 11474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:42301 every 8 connection(s)
I20260807 08:45:23.814085 11402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:45:23.815167 11475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:23.818145 11475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d: Bootstrap starting.
I20260807 08:45:23.818802 11475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:23.819125 11475 log.cc:824] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:23.819998 11475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d: No bootstrap required, opened a new log
I20260807 08:45:23.820734 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 11402
I20260807 08:45:23.820875 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:45:23.821131 29106 external_mini_cluster.cc:1428] Setting key 8eb179260bf9b7551ce19b3114e5f778
I20260807 08:45:23.822379 11475 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5df6d7850d4112b0ab28624d02be9d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42301 } }
I20260807 08:45:23.822594 11475 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:23.822642 11475 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5df6d7850d4112b0ab28624d02be9d, State: Initialized, Role: FOLLOWER
I20260807 08:45:23.822866 11475 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5df6d7850d4112b0ab28624d02be9d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42301 } }
I20260807 08:45:23.822986 11475 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:45:23.823028 11475 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:45:23.823084 11475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:23.824313 11475 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5df6d7850d4112b0ab28624d02be9d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42301 } }
I20260807 08:45:23.824432 11475 leader_election.cc:302] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb5df6d7850d4112b0ab28624d02be9d; no voters:
I20260807 08:45:23.824657 11475 leader_election.cc:288] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:45:23.824716 11480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:45:23.824899 11480 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [term 1 LEADER]: Becoming Leader. State: Replica: cb5df6d7850d4112b0ab28624d02be9d, State: Running, Role: LEADER
I20260807 08:45:23.825007 11475 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:45:23.825074 11480 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5df6d7850d4112b0ab28624d02be9d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42301 } }
I20260807 08:45:23.825716 11481 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb5df6d7850d4112b0ab28624d02be9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5df6d7850d4112b0ab28624d02be9d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42301 } } }
I20260807 08:45:23.825729 11482 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb5df6d7850d4112b0ab28624d02be9d. Latest consensus state: current_term: 1 leader_uuid: "cb5df6d7850d4112b0ab28624d02be9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5df6d7850d4112b0ab28624d02be9d" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42301 } } }
I20260807 08:45:23.825901 11481 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:23.825911 11482 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:23.826225 11489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:45:23.827181 11489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:45:23.829247 11489 catalog_manager.cc:1347] Generated new cluster ID: e6a7f64411b744ce92f1cc65b4844b1a
I20260807 08:45:23.829309 11489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:45:23 dist-test-slave-9zdj krb5kdc[11389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:23.835394 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:23.821802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.28.108.190:42301)
0807 08:45:23.822046 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:23.822057 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:23.822088 (+ 31us) server_negotiation.cc:407] Connection header received
0807 08:45:23.822344 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:23.822352 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:23.822414 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:23.822557 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:23.823574 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:23.825007 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:23.826514 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:23.827272 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:23.827408 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:23.830750 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:23.830755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:23.830755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:23.830771 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:23.832796 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:23.833278 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:23.833282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:23.833282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:23.833933 (+ 651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:23.834205 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:23.834206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:23.834206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:23.834662 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:23.834900 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:23.834995 (+ 95us) server_negotiation.cc:299] Negotiation successful
0807 08:45:23.835176 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":140,"threads_started":1}
I20260807 08:45:23.836550 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:45:23.848098 11489 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:45:23.848875 11489 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:45:23.865568 11489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb5df6d7850d4112b0ab28624d02be9d: Generated new TSK 0
I20260807 08:45:23.865926 11489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:45:23.891538 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 11402
2026-08-07T08:45:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1571 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:45:23 dist-test-slave-9zdj krb5kdc[11504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:45:23 dist-test-slave-9zdj krb5kdc[11504](info): set up 2 sockets
Aug 07 08:45:23 dist-test-slave-9zdj krb5kdc[11504](info): commencing operation
krb5kdc: starting...
W20260807 08:45:25.038270 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:45:25 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:45:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:45:25.207677 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:46821
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:25.307614 11517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:25.307902 11517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:25.307933 11517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:25.309552 11517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:45:25.309702 11517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:25.309736 11517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:25.309752 11517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:45:25.309767 11517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:45:25.311748 11517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:46821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:25.312104 11517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:25.312405 11517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:45:25.318888 11517 server_base.cc:1034] running on GCE node
W20260807 08:45:25.318864 11523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:25.318861 11525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:25.318836 11522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:25.319540 11517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:25.319888 11517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:25.321092 11517 hybrid_clock.cc:648] HybridClock initialized: now 1786092325320992 us; error 119 us; skew 500 ppm
Aug 07 08:45:25 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:25.324774 11517 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:45:25.325436 11517 webserver.cc:460] Webserver started at http://127.28.108.190:45011/ using document root <none> and password file <none>
I20260807 08:45:25.325877 11517 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:25.325974 11517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:25.326169 11517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:25.327527 11517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "3646fe2eb3bb41918de1e475d719be14"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "3a52b7f9d07bb82ab22f9ac08efaeede"
server_key_iv: "87622244fa7ebb9bec758c616e269e6a"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.327984 11517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "3646fe2eb3bb41918de1e475d719be14"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "3a52b7f9d07bb82ab22f9ac08efaeede"
server_key_iv: "87622244fa7ebb9bec758c616e269e6a"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.329795 11517 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260807 08:45:25.330946 11533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:25.331271 11517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:25.331429 11517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "3646fe2eb3bb41918de1e475d719be14"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "3a52b7f9d07bb82ab22f9ac08efaeede"
server_key_iv: "87622244fa7ebb9bec758c616e269e6a"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.331529 11517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:25.352376 11517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:25.355708 11517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:25.355958 11517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:25.360822 11517 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:46821
I20260807 08:45:25.360867 11589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:46821 every 8 connection(s)
I20260807 08:45:25.361320 11517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:45:25.362814 11590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:25.365809 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 11517
I20260807 08:45:25.365998 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:45:25.366179 11590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Bootstrap starting.
I20260807 08:45:25.366392 29106 external_mini_cluster.cc:1428] Setting key 10789dd3fa5192009805b0eaa4d0c4f4
I20260807 08:45:25.367105 11590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:25.367516 11590 log.cc:824] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:25.368409 11590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: No bootstrap required, opened a new log
I20260807 08:45:25.370863 11590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:25.371102 11590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:25.371165 11590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3646fe2eb3bb41918de1e475d719be14, State: Initialized, Role: FOLLOWER
I20260807 08:45:25.371385 11590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:25.371510 11590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:45:25.371567 11590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:45:25.371624 11590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:25.372411 11590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:25.372581 11590 leader_election.cc:302] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3646fe2eb3bb41918de1e475d719be14; no voters:
I20260807 08:45:25.372805 11590 leader_election.cc:288] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:45:25.372869 11595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:45:25.373136 11595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 1 LEADER]: Becoming Leader. State: Replica: 3646fe2eb3bb41918de1e475d719be14, State: Running, Role: LEADER
I20260807 08:45:25.373198 11590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:45:25.373261 11595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:25.373796 11597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3646fe2eb3bb41918de1e475d719be14. Latest consensus state: current_term: 1 leader_uuid: "3646fe2eb3bb41918de1e475d719be14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } } }
I20260807 08:45:25.373802 11596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3646fe2eb3bb41918de1e475d719be14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } } }
I20260807 08:45:25.374022 11596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:25.374329 11597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:25.374549 11604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 07 08:45:25 dist-test-slave-9zdj krb5kdc[11504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:25.375526 11604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:45:25.377769 11604 catalog_manager.cc:1347] Generated new cluster ID: 8e14d6eb684d446b8b8904ebb163868a
I20260807 08:45:25.377837 11604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:45:25.380769 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:25.367098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.28.108.190:46821)
0807 08:45:25.367378 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:25.367388 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:25.367419 (+ 31us) server_negotiation.cc:407] Connection header received
0807 08:45:25.368037 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:25.368046 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:25.368107 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:25.368295 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:25.369159 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.370300 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.371296 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.371972 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.372122 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:25.376242 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:25.376248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:25.376249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:25.376265 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:25.378743 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:25.379234 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:25.379238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:25.379238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:25.379378 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:25.379620 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:25.379621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:25.379621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:25.380012 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:25.380225 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:25.380317 (+ 92us) server_negotiation.cc:299] Negotiation successful
0807 08:45:25.380503 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":132,"threads_started":1}
I20260807 08:45:25.385885 11604 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:45:25.387137 11604 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:45:25.400308 11604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Generated new TSK 0
I20260807 08:45:25.400579 11604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:45:25.442517 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:46821
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:25.534588 11618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:25.534843 11618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:25.534914 11618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:25.536315 11618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:25.536374 11618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:25.536405 11618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:45:25.538173 11618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:25.538466 11618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:25.538759 11618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:25.544871 11628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:25.544884 11625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:25.544919 11618 server_base.cc:1034] running on GCE node
W20260807 08:45:25.544864 11624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:25.545444 11618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:25.545826 11618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:25.547025 11618 hybrid_clock.cc:648] HybridClock initialized: now 1786092325546990 us; error 62 us; skew 500 ppm
Aug 07 08:45:25 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:25.555544 11618 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:45:25.556268 11618 webserver.cc:460] Webserver started at http://127.28.108.129:36851/ using document root <none> and password file <none>
I20260807 08:45:25.556584 11618 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:25.556679 11618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:25.556879 11618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:25.558297 11618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "75d2b3ec34fb4ca8b33e8a5476510841"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "6a242e7eeaa41effea06aa1ecdb5959d"
server_key_iv: "b5128c241312f374288c9f351bf23f21"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.558740 11618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "75d2b3ec34fb4ca8b33e8a5476510841"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "6a242e7eeaa41effea06aa1ecdb5959d"
server_key_iv: "b5128c241312f374288c9f351bf23f21"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.560617 11618 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 08:45:25.561836 11635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:25.562062 11618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:25.562213 11618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "75d2b3ec34fb4ca8b33e8a5476510841"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "6a242e7eeaa41effea06aa1ecdb5959d"
server_key_iv: "b5128c241312f374288c9f351bf23f21"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.562317 11618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:25.576442 11618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:25.579782 11618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:25.580009 11618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:25.580322 11618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:25.580729 11618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:25.580801 11618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:25.580839 11618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:25.580869 11618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:25.588068 11618 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:38665
I20260807 08:45:25.588109 11760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:38665 every 8 connection(s)
I20260807 08:45:25.588493 11618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:45:25.590447 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 11618
I20260807 08:45:25.590584 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:45:25.590838 29106 external_mini_cluster.cc:1428] Setting key 400e0454c08e34d5c02c8034e79fbfb7
Aug 07 08:45:25 dist-test-slave-9zdj krb5kdc[11504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:25.601382 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:25.589440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:35093 (local address 127.28.108.190:46821)
0807 08:45:25.589517 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:25.589520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:25.589583 (+ 63us) server_negotiation.cc:407] Connection header received
0807 08:45:25.590247 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:25.590249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:25.590265 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:25.590341 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:25.591790 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.592400 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.594038 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.594407 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.594500 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:25.597720 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:25.597774 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:25.597774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:25.597784 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:25.599527 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:25.600127 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:25.600129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:25.600129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:25.600226 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:25.600561 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:25.600562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:25.600562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:25.600709 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:25.600809 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:25.601152 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 08:45:25.601271 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 08:45:25.601653 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:25.589259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:35093)
0807 08:45:25.589440 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:25.589489 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:25.590079 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:25.590469 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:25.590475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:25.591380 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:25.591648 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:25.591656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.592517 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.592519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:25.593923 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:25.593927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.594517 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.594518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:25.594636 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:25.595322 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:25.595334 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:25.597473 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:25.599670 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:25.599682 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:25.599685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:25.600022 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:25.600318 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:25.600319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:25.600320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:25.600477 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:25.600799 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:25.600808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:25.600993 (+ 185us) client_negotiation.cc:769] Sending connection context
0807 08:45:25.601115 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 08:45:25.601332 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
I20260807 08:45:25.602397 11761 heartbeater.cc:347] Connected to a master server at 127.28.108.190:46821
I20260807 08:45:25.602533 11761 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:25.602857 11761 heartbeater.cc:511] Master 127.28.108.190:46821 requested a full tablet report, sending...
I20260807 08:45:25.603953 11551 ts_manager.cc:194] Registered new tserver with Master: 75d2b3ec34fb4ca8b33e8a5476510841 (127.28.108.129:38665)
I20260807 08:45:25.604630 11551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:35093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:45:25.630311 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:46821
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:25.723609 11767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:25.723934 11767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:25.723963 11767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:25.725406 11767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:25.725485 11767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:25.725519 11767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:45:25.727658 11767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:25.728056 11767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:25.728340 11767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:25.734422 11775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:25.734427 11772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:25.734570 11773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:25.734705 11767 server_base.cc:1034] running on GCE node
I20260807 08:45:25.735461 11767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:25.736027 11767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:25.737457 11767 hybrid_clock.cc:648] HybridClock initialized: now 1786092325737381 us; error 98 us; skew 500 ppm
Aug 07 08:45:25 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:25.740908 11767 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:45:25.741483 11767 webserver.cc:460] Webserver started at http://127.28.108.130:34163/ using document root <none> and password file <none>
I20260807 08:45:25.741845 11767 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:25.741928 11767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:25.742084 11767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:25.743301 11767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "b20980151bed4bc0a3b6a1139fe4d03c"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "1f583d6e956b6d379624cf4310fc44f7"
server_key_iv: "e0bc4f115d491f0899df1a457191d9b5"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.743772 11767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "b20980151bed4bc0a3b6a1139fe4d03c"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "1f583d6e956b6d379624cf4310fc44f7"
server_key_iv: "e0bc4f115d491f0899df1a457191d9b5"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.745514 11767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260807 08:45:25.746526 11782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:25.746748 11767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:25.746901 11767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "b20980151bed4bc0a3b6a1139fe4d03c"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "1f583d6e956b6d379624cf4310fc44f7"
server_key_iv: "e0bc4f115d491f0899df1a457191d9b5"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.746996 11767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:25.760284 11767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:25.763247 11767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:25.763464 11767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:25.763758 11767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:25.764128 11767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:25.764199 11767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:25.764245 11767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:25.764271 11767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:25.771684 11767 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:46265
I20260807 08:45:25.771752 11906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:46265 every 8 connection(s)
I20260807 08:45:25.772125 11767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:45:25.778447 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 11767
I20260807 08:45:25.778632 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:45:25.778887 29106 external_mini_cluster.cc:1428] Setting key 35721744bf41471dbc0ee5693ad66edd
Aug 07 08:45:25 dist-test-slave-9zdj krb5kdc[11504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:25.784250 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:25.773077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:56729 (local address 127.28.108.190:46821)
0807 08:45:25.773193 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:25.773195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:25.773542 (+ 347us) server_negotiation.cc:407] Connection header received
0807 08:45:25.774271 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:25.774273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:25.774289 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:25.774362 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:25.775626 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.776206 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.777426 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.777734 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.777830 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:25.780680 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:25.780682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:25.780683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:25.780700 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:25.782357 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:25.782962 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:25.782963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:25.782964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:25.783039 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:25.783355 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:25.783357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:25.783357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:25.783531 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:25.783604 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:25.783985 (+ 381us) server_negotiation.cc:299] Negotiation successful
0807 08:45:25.784098 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 08:45:25.784435 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:25.773024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:56729)
0807 08:45:25.773370 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:25.773418 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:25.774127 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:25.774449 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:25.774454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:25.775256 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:25.775494 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:25.775500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.776350 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.776351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:25.777313 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:25.777316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.777842 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.777843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:25.778017 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:25.778772 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:25.778782 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:25.780556 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:25.782470 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:25.782483 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:25.782486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:25.782864 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:25.783125 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:25.783125 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:25.783126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:25.783264 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:25.783616 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:25.783624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:25.783796 (+ 172us) client_negotiation.cc:769] Sending connection context
0807 08:45:25.783933 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 08:45:25.784125 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":212,"threads_started":1}
I20260807 08:45:25.785091 11907 heartbeater.cc:347] Connected to a master server at 127.28.108.190:46821
I20260807 08:45:25.785251 11907 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:25.785503 11907 heartbeater.cc:511] Master 127.28.108.190:46821 requested a full tablet report, sending...
I20260807 08:45:25.786113 11551 ts_manager.cc:194] Registered new tserver with Master: b20980151bed4bc0a3b6a1139fe4d03c (127.28.108.130:46265)
I20260807 08:45:25.786713 11551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:56729
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:45:25.814875 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:46821
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:25.914873 11912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:25.915122 11912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:25.915153 11912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:25.916667 11912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:25.916734 11912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:25.916771 11912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:45:25.918687 11912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:25.918962 11912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:25.919195 11912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:25.925537 11920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:25.925529 11918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:25.925817 11912 server_base.cc:1034] running on GCE node
W20260807 08:45:25.925545 11917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:25.926126 11912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:25.926498 11912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:25.927660 11912 hybrid_clock.cc:648] HybridClock initialized: now 1786092325927639 us; error 38 us; skew 500 ppm
Aug 07 08:45:25 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:25.930745 11912 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:45:25.931366 11912 webserver.cc:460] Webserver started at http://127.28.108.131:38333/ using document root <none> and password file <none>
I20260807 08:45:25.931633 11912 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:25.931708 11912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:25.931887 11912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:25.933090 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "7081c41bf91d45dd9f00efbb5f9e954e"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "05be02bf9437887949e320ab2ea6dc44"
server_key_iv: "a1a3ed95a839b3b34c5cb071c8c5b9d9"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.933665 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "7081c41bf91d45dd9f00efbb5f9e954e"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "05be02bf9437887949e320ab2ea6dc44"
server_key_iv: "a1a3ed95a839b3b34c5cb071c8c5b9d9"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.935536 11912 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:45:25.936718 11928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:25.937039 11912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:25.937327 11912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "7081c41bf91d45dd9f00efbb5f9e954e"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "05be02bf9437887949e320ab2ea6dc44"
server_key_iv: "a1a3ed95a839b3b34c5cb071c8c5b9d9"
server_key_version: "encryptionkey@0"
I20260807 08:45:25.937454 11912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:25.956467 11912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:25.959235 11912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:25.959546 11912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:25.959839 11912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:25.960222 11912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:25.960296 11912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:25.960346 11912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:25.960373 11912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:25.967137 11912 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:42437
I20260807 08:45:25.967183 12056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:42437 every 8 connection(s)
I20260807 08:45:25.967586 11912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:45:25.972599 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 11912
I20260807 08:45:25.972834 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:45:25.973033 29106 external_mini_cluster.cc:1428] Setting key 2f942895be1da25363c90a81048cf66e
Aug 07 08:45:25 dist-test-slave-9zdj krb5kdc[11504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:25.979714 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:25.968479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:48797 (local address 127.28.108.190:46821)
0807 08:45:25.968555 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:25.968557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:25.968760 (+ 203us) server_negotiation.cc:407] Connection header received
0807 08:45:25.969422 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:25.969424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:25.969439 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:25.969505 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:25.970705 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.971239 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.972338 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.972579 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.972844 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:25.976046 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:25.976047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:25.976048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:25.976056 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:25.977767 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:25.978409 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:25.978411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:25.978411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:25.978481 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:25.978862 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:25.978863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:25.978863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:25.978990 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:25.979052 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:25.979529 (+ 477us) server_negotiation.cc:299] Negotiation successful
0807 08:45:25.979629 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 08:45:25.979935 12059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:25.968455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:48797)
0807 08:45:25.968634 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:25.968675 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:25.969197 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:25.969601 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:25.969605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:25.970368 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:25.970587 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:25.970592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.971351 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.971352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:25.972217 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:25.972219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.972875 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.972876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:25.973058 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:25.974040 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:25.974052 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:25.975903 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:25.977866 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:25.977876 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:25.977878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:25.978150 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:25.978593 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:25.978594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:25.978594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:25.978781 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:25.979057 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:25.979088 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:25.979341 (+ 253us) client_negotiation.cc:769] Sending connection context
0807 08:45:25.979458 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 08:45:25.979635 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260807 08:45:25.980629 12057 heartbeater.cc:347] Connected to a master server at 127.28.108.190:46821
I20260807 08:45:25.980813 12057 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:25.980999 12057 heartbeater.cc:511] Master 127.28.108.190:46821 requested a full tablet report, sending...
I20260807 08:45:25.981539 11551 ts_manager.cc:194] Registered new tserver with Master: 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437)
I20260807 08:45:25.982079 11551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:48797
I20260807 08:45:25.985880 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:45:25.994921 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:25.987330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.28.108.190:46821)
0807 08:45:25.987420 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:25.987422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:25.987634 (+ 212us) server_negotiation.cc:407] Connection header received
0807 08:45:25.987735 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:25.987736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:25.987747 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:25.987830 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:25.988549 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.989123 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.990045 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:25.990275 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:25.990374 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:25.991765 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:25.991767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:25.991768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:25.991777 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:25.993447 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:25.993941 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:25.993942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:25.993943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:25.994011 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:25.994250 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:25.994251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:25.994252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:25.994442 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:25.994564 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:25.994703 (+ 139us) server_negotiation.cc:299] Negotiation successful
0807 08:45:25.994808 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 08:45:25.997138 11551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:45:25.999282 11551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:45:26.007265 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.002697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.28.108.131:42437)
0807 08:45:26.003351 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.003355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.003365 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:45:26.003401 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.003403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.003419 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.003491 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:26.003653 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.004870 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.006202 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.006757 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.006870 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.006942 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.007113 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.007161 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":104,"threads_started":1}
I20260807 08:45:26.007253 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.002608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:42437 (local address 127.0.0.1:55534)
0807 08:45:26.002802 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.002848 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:26.002922 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:26.003505 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:26.003506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:26.003508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:26.003562 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.003563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.004994 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.004995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.006093 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.006095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.006870 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.006871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.006969 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.006992 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 08:45:26.007089 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 08:45:26.007164 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260807 08:45:26.007450 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.002686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:38665 (local address 127.0.0.1:35038)
0807 08:45:26.003105 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.003130 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:26.003183 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:26.003487 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:26.003488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:26.003490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:26.003550 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.003552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.004552 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.004553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.005959 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.005962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.007073 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.007074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.007134 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.007139 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:45:26.007188 (+ 49us) client_negotiation.cc:240] Negotiation successful
0807 08:45:26.007237 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":183,"threads_started":1}
I20260807 08:45:26.007524 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.003029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.28.108.129:38665)
0807 08:45:26.003233 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.003238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.003248 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:45:26.003298 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.003300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.003317 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.003398 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:26.003640 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.004441 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.006059 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.006893 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.006977 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.007037 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.007387 (+ 350us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.007436 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":124,"threads_started":1}
I20260807 08:45:26.008292 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.002686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:46265 (local address 127.0.0.1:59702)
0807 08:45:26.002911 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.002922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:26.002966 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:26.003311 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:26.003315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:26.003317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:26.003407 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.003410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.005677 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.005679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.006893 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.006896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.008011 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.008012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.008080 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.008084 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:45:26.008177 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 08:45:26.008222 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":96,"threads_started":1}
I20260807 08:45:26.008390 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.002940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.28.108.130:46265)
0807 08:45:26.003098 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.003103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.003117 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:45:26.003163 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.003165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.003184 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.003296 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:26.003733 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.004860 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.007007 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.007876 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.007996 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.008060 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.008204 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.008255 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260807 08:45:26.008526 11688 tablet_service.cc:1467] Processing CreateTablet for tablet d05a7503cc1948159c735d0c6cc81e10 (DEFAULT_TABLE table=test-table [id=96beaff97dcf4f4697f3cfcb24d35067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:26.008940 11688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05a7503cc1948159c735d0c6cc81e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:26.009126 11833 tablet_service.cc:1467] Processing CreateTablet for tablet d05a7503cc1948159c735d0c6cc81e10 (DEFAULT_TABLE table=test-table [id=96beaff97dcf4f4697f3cfcb24d35067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:26.009425 11833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05a7503cc1948159c735d0c6cc81e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:26.010426 11982 tablet_service.cc:1467] Processing CreateTablet for tablet d05a7503cc1948159c735d0c6cc81e10 (DEFAULT_TABLE table=test-table [id=96beaff97dcf4f4697f3cfcb24d35067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:26.010823 11982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05a7503cc1948159c735d0c6cc81e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:26.011965 12076 tablet_bootstrap.cc:492] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Bootstrap starting.
I20260807 08:45:26.012672 12076 tablet_bootstrap.cc:654] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:26.014010 12076 log.cc:824] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:26.014452 12077 tablet_bootstrap.cc:492] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: Bootstrap starting.
I20260807 08:45:26.014920 12076 tablet_bootstrap.cc:492] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: No bootstrap required, opened a new log
I20260807 08:45:26.015026 12076 ts_tablet_manager.cc:1397] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:45:26.015267 12077 tablet_bootstrap.cc:654] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:26.015831 12079 tablet_bootstrap.cc:492] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: Bootstrap starting.
I20260807 08:45:26.015936 12077 log.cc:824] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:26.016661 12079 tablet_bootstrap.cc:654] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:26.016767 12077 tablet_bootstrap.cc:492] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: No bootstrap required, opened a new log
I20260807 08:45:26.016887 12077 ts_tablet_manager.cc:1397] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:45:26.017338 12079 log.cc:824] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:26.017287 12076 raft_consensus.cc:348] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } }
I20260807 08:45:26.017496 12076 raft_consensus.cc:374] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:26.017552 12076 raft_consensus.cc:729] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d2b3ec34fb4ca8b33e8a5476510841, State: Initialized, Role: FOLLOWER
I20260807 08:45:26.017781 12076 consensus_queue.cc:260] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } }
I20260807 08:45:26.018121 12076 ts_tablet_manager.cc:1428] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:45:26.018132 11761 heartbeater.cc:503] Master 127.28.108.190:46821 was elected leader, sending a full tablet report...
I20260807 08:45:26.018966 12079 tablet_bootstrap.cc:492] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: No bootstrap required, opened a new log
I20260807 08:45:26.019055 12079 ts_tablet_manager.cc:1397] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:45:26.018940 12077 raft_consensus.cc:348] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } }
I20260807 08:45:26.019140 12077 raft_consensus.cc:374] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:26.019189 12077 raft_consensus.cc:729] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20980151bed4bc0a3b6a1139fe4d03c, State: Initialized, Role: FOLLOWER
I20260807 08:45:26.019346 12077 consensus_queue.cc:260] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } }
I20260807 08:45:26.019656 11907 heartbeater.cc:503] Master 127.28.108.190:46821 was elected leader, sending a full tablet report...
I20260807 08:45:26.019654 12077 ts_tablet_manager.cc:1428] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:45:26.021152 12079 raft_consensus.cc:348] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } }
I20260807 08:45:26.021343 12079 raft_consensus.cc:374] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:26.021394 12079 raft_consensus.cc:729] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7081c41bf91d45dd9f00efbb5f9e954e, State: Initialized, Role: FOLLOWER
I20260807 08:45:26.021521 12079 consensus_queue.cc:260] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } }
I20260807 08:45:26.021893 12079 ts_tablet_manager.cc:1428] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:45:26.021951 12057 heartbeater.cc:503] Master 127.28.108.190:46821 was elected leader, sending a full tablet report...
W20260807 08:45:26.022739 11908 tablet.cc:2369] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:45:26.089520 11762 tablet.cc:2369] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:45:26.188004 12084 raft_consensus.cc:482] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:45:26.188194 12084 raft_consensus.cc:504] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } }
I20260807 08:45:26.188666 12084 leader_election.cc:288] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437), 75d2b3ec34fb4ca8b33e8a5476510841 (127.28.108.129:38665)
I20260807 08:45:26.192546 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.188972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:38665 (local address 127.28.108.130:56789)
0807 08:45:26.189097 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.189109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:26.189178 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:26.189390 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:26.189391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:26.189393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:26.189472 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.189475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.190310 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.190311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.191298 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.191301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.192184 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.192185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.192315 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.192319 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:45:26.192384 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 08:45:26.192441 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260807 08:45:26.192552 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.189058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:56789 (local address 127.28.108.129:38665)
0807 08:45:26.189113 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.189115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.189232 (+ 117us) server_negotiation.cc:407] Connection header received
0807 08:45:26.189269 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.189270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.189281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.189361 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:26.189566 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.190133 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.191424 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.192062 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.192176 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.192189 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.192417 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.192469 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260807 08:45:26.193085 11711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05a7503cc1948159c735d0c6cc81e10" candidate_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" is_pre_election: true
I20260807 08:45:26.193140 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.188971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:42437 (local address 127.28.108.130:48951)
0807 08:45:26.189202 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.189214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:26.189299 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:26.189479 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:26.189480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:26.189481 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:26.189583 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.189585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.190659 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.190660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.191737 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.191740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.192812 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.192813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.192904 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.192915 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 08:45:26.192998 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 08:45:26.193043 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":20,"thread_start_us":137,"threads_started":1}
I20260807 08:45:26.193125 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.189153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:48951 (local address 127.28.108.131:42437)
0807 08:45:26.189200 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.189202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.189289 (+ 87us) server_negotiation.cc:407] Connection header received
0807 08:45:26.189382 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.189383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.189395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.189458 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:26.189837 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.190532 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.191883 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.192572 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.192738 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.192751 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.193006 (+ 255us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.193054 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 08:45:26.193295 11711 raft_consensus.cc:2454] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20980151bed4bc0a3b6a1139fe4d03c in term 0.
I20260807 08:45:26.193569 11783 leader_election.cc:302] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75d2b3ec34fb4ca8b33e8a5476510841, b20980151bed4bc0a3b6a1139fe4d03c; no voters:
I20260807 08:45:26.193547 12004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05a7503cc1948159c735d0c6cc81e10" candidate_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" is_pre_election: true
I20260807 08:45:26.193756 12004 raft_consensus.cc:2454] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20980151bed4bc0a3b6a1139fe4d03c in term 0.
I20260807 08:45:26.193846 12084 raft_consensus.cc:2781] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:45:26.193955 12084 raft_consensus.cc:482] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:45:26.194022 12084 raft_consensus.cc:3037] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:26.194931 12084 raft_consensus.cc:504] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } }
I20260807 08:45:26.195122 12084 leader_election.cc:288] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [CANDIDATE]: Term 1 election: Requested vote from peers 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437), 75d2b3ec34fb4ca8b33e8a5476510841 (127.28.108.129:38665)
I20260807 08:45:26.195322 12004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05a7503cc1948159c735d0c6cc81e10" candidate_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7081c41bf91d45dd9f00efbb5f9e954e"
I20260807 08:45:26.195444 12004 raft_consensus.cc:3037] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:26.195559 11711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05a7503cc1948159c735d0c6cc81e10" candidate_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75d2b3ec34fb4ca8b33e8a5476510841"
I20260807 08:45:26.195668 11711 raft_consensus.cc:3037] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:26.196604 12004 raft_consensus.cc:2454] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20980151bed4bc0a3b6a1139fe4d03c in term 1.
I20260807 08:45:26.196801 11711 raft_consensus.cc:2454] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20980151bed4bc0a3b6a1139fe4d03c in term 1.
I20260807 08:45:26.196964 11785 leader_election.cc:302] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7081c41bf91d45dd9f00efbb5f9e954e, b20980151bed4bc0a3b6a1139fe4d03c; no voters:
I20260807 08:45:26.197157 12084 raft_consensus.cc:2781] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:45:26.197377 12084 raft_consensus.cc:686] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 LEADER]: Becoming Leader. State: Replica: b20980151bed4bc0a3b6a1139fe4d03c, State: Running, Role: LEADER
I20260807 08:45:26.197510 12084 consensus_queue.cc:237] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } }
I20260807 08:45:26.198724 11551 catalog_manager.cc:5495] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c reported cstate change: term changed from 0 to 1, leader changed from <none> to b20980151bed4bc0a3b6a1139fe4d03c (127.28.108.130). New cstate: current_term: 1 leader_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } health_report { overall_health: HEALTHY } } }
W20260807 08:45:26.218379 12058 tablet.cc:2369] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:45:26.243953 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.240443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.28.108.130:46265)
0807 08:45:26.240536 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.240538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.240547 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:45:26.240596 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.240596 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.240617 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.240721 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:45:26.241050 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.241785 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.242830 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.243042 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.243106 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.243454 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:45:26.243588 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:45:26.243667 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.243812 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.243841 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 08:45:26.246438 11711 raft_consensus.cc:1261] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Refusing update from remote peer b20980151bed4bc0a3b6a1139fe4d03c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:45:26.246433 12004 raft_consensus.cc:1261] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e [term 1 FOLLOWER]: Refusing update from remote peer b20980151bed4bc0a3b6a1139fe4d03c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:45:26.246901 12087 consensus_queue.cc:1035] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:45:26.247164 12084 consensus_queue.cc:1035] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:45:26.250769 12093 mvcc.cc:205] Tried to move back new op lower bound from 7315834168301416448 to 7315834168105639936. Current Snapshot: MvccSnapshot[applied={T|T < 7315834168301416448}]
I20260807 08:45:26.251031 12092 mvcc.cc:205] Tried to move back new op lower bound from 7315834168301416448 to 7315834168105639936. Current Snapshot: MvccSnapshot[applied={T|T < 7315834168301416448}]
I20260807 08:45:26.253985 11550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:45:26.254112 11550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:45:26.255316 11550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Sending DeleteTablet for 3 replicas of tablet d05a7503cc1948159c735d0c6cc81e10
I20260807 08:45:26.255824 11982 tablet_service.cc:1514] Processing DeleteTablet for tablet d05a7503cc1948159c735d0c6cc81e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:26 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:55534
I20260807 08:45:26.255846 11833 tablet_service.cc:1514] Processing DeleteTablet for tablet d05a7503cc1948159c735d0c6cc81e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:26 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:59702
I20260807 08:45:26.255901 11688 tablet_service.cc:1514] Processing DeleteTablet for tablet d05a7503cc1948159c735d0c6cc81e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:26 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:35038
I20260807 08:45:26.256122 12101 tablet_replica.cc:330] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: stopping tablet replica
I20260807 08:45:26.256256 12101 raft_consensus.cc:2229] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:45:26.256307 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 11618
I20260807 08:45:26.256384 12101 raft_consensus.cc:2258] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:45:26.259166 12101 ts_tablet_manager.cc:1905] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:45:26.261034 12103 tablet_replica.cc:330] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: stopping tablet replica
I20260807 08:45:26.261250 12103 raft_consensus.cc:2229] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:45:26.261420 12103 raft_consensus.cc:2258] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:45:26.261955 12103 ts_tablet_manager.cc:1905] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:45:26.262501 12101 ts_tablet_manager.cc:1918] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:45:26.262615 12101 log.cc:1196] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/d05a7503cc1948159c735d0c6cc81e10
I20260807 08:45:26.262943 12101 ts_tablet_manager.cc:1939] T d05a7503cc1948159c735d0c6cc81e10 P 7081c41bf91d45dd9f00efbb5f9e954e: Deleting consensus metadata
I20260807 08:45:26.263540 11536 catalog_manager.cc:4841] TS 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437): tablet d05a7503cc1948159c735d0c6cc81e10 (table test-table [id=96beaff97dcf4f4697f3cfcb24d35067]) successfully deleted
I20260807 08:45:26.263947 12103 ts_tablet_manager.cc:1918] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:45:26.264055 12103 log.cc:1196] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/d05a7503cc1948159c735d0c6cc81e10
I20260807 08:45:26.264385 12103 ts_tablet_manager.cc:1939] T d05a7503cc1948159c735d0c6cc81e10 P b20980151bed4bc0a3b6a1139fe4d03c: Deleting consensus metadata
I20260807 08:45:26.264992 11537 catalog_manager.cc:4841] TS b20980151bed4bc0a3b6a1139fe4d03c (127.28.108.130:46265): tablet d05a7503cc1948159c735d0c6cc81e10 (table test-table [id=96beaff97dcf4f4697f3cfcb24d35067]) successfully deleted
W20260807 08:45:26.267663 11534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:38665 (error 108)
I20260807 08:45:26.268364 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.268202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:38665 (local address 127.0.0.1:35054)
0807 08:45:26.268274 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.268322 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:38665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 08:45:26.268464 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 11767
W20260807 08:45:26.272874 11534 catalog_manager.cc:4584] TS 75d2b3ec34fb4ca8b33e8a5476510841 (127.28.108.129:38665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d05a7503cc1948159c735d0c6cc81e10: Network error: Client connection negotiation failed: client connection to 127.28.108.129:38665: connect: Connection refused (error 111)
I20260807 08:45:26.278131 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 11912
I20260807 08:45:26.286981 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 11517
I20260807 08:45:26.296885 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:46821
--webserver_interface=127.28.108.190
--webserver_port=45011
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:26.390362 12105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:26.390695 12105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:26.390746 12105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:26.392318 12105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:45:26.392426 12105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:26.392454 12105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:26.392479 12105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:45:26.392518 12105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:45:26.394601 12105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:46821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=45011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:26.395020 12105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:26.395334 12105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:26.407902 12111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:26.407823 12114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:26.407878 12105 server_base.cc:1034] running on GCE node
W20260807 08:45:26.407824 12112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:26.408598 12105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:26.409047 12105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:26.410250 12105 hybrid_clock.cc:648] HybridClock initialized: now 1786092326410223 us; error 50 us; skew 500 ppm
Aug 07 08:45:26 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:26.414209 12105 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:45:26.415086 12105 webserver.cc:460] Webserver started at http://127.28.108.190:45011/ using document root <none> and password file <none>
I20260807 08:45:26.415537 12105 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:26.415666 12105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:26.417541 12105 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260807 08:45:26.418740 12122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:26.419087 12105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:26.419364 12105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "3646fe2eb3bb41918de1e475d719be14"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "3a52b7f9d07bb82ab22f9ac08efaeede"
server_key_iv: "87622244fa7ebb9bec758c616e269e6a"
server_key_version: "encryptionkey@0"
I20260807 08:45:26.419953 12105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:26.430513 12105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:26.433797 12105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:26.433996 12105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:26.438793 12105 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:46821
I20260807 08:45:26.438845 12180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:46821 every 8 connection(s)
I20260807 08:45:26.439329 12105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:45:26.441532 12181 sys_catalog.cc:263] Verifying existing consensus state
I20260807 08:45:26.442448 12181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Bootstrap starting.
I20260807 08:45:26.444582 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 12105
I20260807 08:45:26.445079 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:38665
--local_ip_for_outbound_sockets=127.28.108.129
--tserver_master_addrs=127.28.108.190:46821
--webserver_port=36851
--webserver_interface=127.28.108.129
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:45:26.454173 12181 log.cc:824] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:26.456719 12181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:45:26.457104 12181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Bootstrap complete.
I20260807 08:45:26.459857 12181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:26.460173 12181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3646fe2eb3bb41918de1e475d719be14, State: Initialized, Role: FOLLOWER
I20260807 08:45:26.460434 12181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:26.460541 12181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:45:26.460635 12181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:45:26.460685 12181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:45:26.462036 12181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:26.462147 12181 leader_election.cc:302] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3646fe2eb3bb41918de1e475d719be14; no voters:
I20260807 08:45:26.462414 12181 leader_election.cc:288] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 08:45:26.462459 12186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:45:26.462618 12186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 2 LEADER]: Becoming Leader. State: Replica: 3646fe2eb3bb41918de1e475d719be14, State: Running, Role: LEADER
I20260807 08:45:26.462736 12186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:26.462742 12181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:45:26.463074 12187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3646fe2eb3bb41918de1e475d719be14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } } }
I20260807 08:45:26.463268 12187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:26.463735 12192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:45:26.464596 12188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3646fe2eb3bb41918de1e475d719be14. Latest consensus state: current_term: 2 leader_uuid: "3646fe2eb3bb41918de1e475d719be14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } } }
I20260807 08:45:26.464699 12188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:26.464776 12192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:45:26.464895 12192 catalog_manager.cc:1259] Loaded cluster ID: 8e14d6eb684d446b8b8904ebb163868a
I20260807 08:45:26.464938 12192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:45:26.465234 12192 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 08:45:26.470180 12192 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 08:45:26.473380 12192 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:45:26.473753 12192 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Loaded TSK: 0
I20260807 08:45:26.474031 12192 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 08:45:26.544710 12183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:26.545009 12183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:26.545055 12183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:26.546528 12183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:26.546625 12183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:26.546671 12183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:45:26.548395 12183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:38665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=36851
--tserver_master_addrs=127.28.108.190:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:26.548748 12183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:26.549033 12183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:26.555112 12210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:26.555122 12213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:26.555119 12211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:26.556103 12183 server_base.cc:1034] running on GCE node
I20260807 08:45:26.556457 12183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:26.556773 12183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:26.557951 12183 hybrid_clock.cc:648] HybridClock initialized: now 1786092326557920 us; error 48 us; skew 500 ppm
Aug 07 08:45:26 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:26.561236 12183 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:45:26.561903 12183 webserver.cc:460] Webserver started at http://127.28.108.129:36851/ using document root <none> and password file <none>
I20260807 08:45:26.562177 12183 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:26.562287 12183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:26.564612 12183 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:45:26.565990 12223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:26.566462 12183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:26.566633 12183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "75d2b3ec34fb4ca8b33e8a5476510841"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "6a242e7eeaa41effea06aa1ecdb5959d"
server_key_iv: "b5128c241312f374288c9f351bf23f21"
server_key_version: "encryptionkey@0"
I20260807 08:45:26.567302 12183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:26.583706 12183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:26.587030 12183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:26.587246 12183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:26.587563 12183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:26.588160 12231 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 08:45:26.590457 12183 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 08:45:26.590592 12183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260807 08:45:26.590643 12183 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 08:45:26.591609 12183 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 08:45:26.591701 12183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260807 08:45:26.591781 12231 tablet_bootstrap.cc:492] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Bootstrap starting.
I20260807 08:45:26.600682 12183 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:38665
I20260807 08:45:26.600888 12348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:38665 every 8 connection(s)
I20260807 08:45:26.601236 12183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:45:26.603410 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 12183
I20260807 08:45:26.603850 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:46265
--local_ip_for_outbound_sockets=127.28.108.130
--tserver_master_addrs=127.28.108.190:46821
--webserver_port=34163
--webserver_interface=127.28.108.130
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:45:26.604779 12231 log.cc:824] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:26.607692 12231 tablet_bootstrap.cc:492] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:45:26.608071 12231 tablet_bootstrap.cc:492] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Bootstrap complete.
I20260807 08:45:26.609504 12231 ts_tablet_manager.cc:1397] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.011s
Aug 07 08:45:26 dist-test-slave-9zdj krb5kdc[11504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:26.611944 12231 raft_consensus.cc:348] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } }
I20260807 08:45:26.612382 12231 raft_consensus.cc:729] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d2b3ec34fb4ca8b33e8a5476510841, State: Initialized, Role: FOLLOWER
I20260807 08:45:26.612596 12231 consensus_queue.cc:260] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } }
I20260807 08:45:26.612979 12231 ts_tablet_manager.cc:1428] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:45:26.618173 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.602384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:56711 (local address 127.28.108.190:46821)
0807 08:45:26.603897 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.604205 (+ 308us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.604281 (+ 76us) server_negotiation.cc:407] Connection header received
0807 08:45:26.604445 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.604453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.604520 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.604678 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:26.605929 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.606916 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.608594 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.609152 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.609217 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.612609 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:26.612615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:26.612615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:26.612641 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:26.615271 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:26.616056 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:26.616062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:26.616063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:26.616226 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:26.616634 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:26.616636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:26.616636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:26.617047 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:26.617122 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.617654 (+ 532us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.617874 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1439,"thread_start_us":111,"threads_started":1}
I20260807 08:45:26.619048 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.602137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:56711)
0807 08:45:26.602348 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.602409 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:26.603021 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:26.604843 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:26.604898 (+ 55us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:26.605397 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:26.605749 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.605759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.607067 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.607069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.608467 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.608471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.609333 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.609334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.609444 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.610120 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:26.610131 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:26.612284 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:26.615472 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:26.615485 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:26.615488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:26.615925 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:26.616325 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:26.616326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:26.616327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:26.616534 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:26.617187 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:26.617197 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:26.617386 (+ 189us) client_negotiation.cc:769] Sending connection context
0807 08:45:26.618627 (+ 1241us) client_negotiation.cc:240] Negotiation successful
0807 08:45:26.618849 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":109,"threads_started":1}
I20260807 08:45:26.619956 12349 heartbeater.cc:347] Connected to a master server at 127.28.108.190:46821
I20260807 08:45:26.620077 12349 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:26.620321 12349 heartbeater.cc:511] Master 127.28.108.190:46821 requested a full tablet report, sending...
I20260807 08:45:26.621197 12141 ts_manager.cc:194] Registered new tserver with Master: 75d2b3ec34fb4ca8b33e8a5476510841 (127.28.108.129:38665)
I20260807 08:45:26.622537 12141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:56711
I20260807 08:45:26.623656 12349 heartbeater.cc:503] Master 127.28.108.190:46821 was elected leader, sending a full tablet report...
Aug 07 08:45:26 dist-test-slave-9zdj krb5kdc[11504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
I20260807 08:45:26.633208 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.621992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:38665 (local address 127.0.0.1:35064)
0807 08:45:26.622327 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.622370 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:26.622458 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:26.622789 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:26.622795 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:26.623149 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:26.623208 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.623211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.624321 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.624322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.625329 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.625334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.626094 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.626095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.626203 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.626870 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:26.626876 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:26.628961 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:26.631326 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:26.631341 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:26.631345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:26.631691 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:26.632022 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:26.632023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:26.632023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:26.632102 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:26.632641 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:26.632653 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:26.632731 (+ 78us) client_negotiation.cc:769] Sending connection context
0807 08:45:26.632906 (+ 175us) client_negotiation.cc:240] Negotiation successful
0807 08:45:26.633086 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":102,"threads_started":1}
I20260807 08:45:26.633219 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.622102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.28.108.129:38665)
0807 08:45:26.622302 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.622310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.622456 (+ 146us) server_negotiation.cc:407] Connection header received
0807 08:45:26.622543 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.622546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.622565 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.622652 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:26.623308 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.624240 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.625461 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.625987 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.626086 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.629111 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:26.629124 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:26.629125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:26.629133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:26.631198 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:26.631838 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:26.631843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:26.631843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:26.631923 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:26.632222 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:26.632224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:26.632224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:26.632552 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:26.632645 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.633009 (+ 364us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.633138 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
I20260807 08:45:26.634248 12275 tablet_service.cc:1514] Processing DeleteTablet for tablet d05a7503cc1948159c735d0c6cc81e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:26 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:35064
I20260807 08:45:26.634992 12358 tablet_replica.cc:330] stopping tablet replica
I20260807 08:45:26.635157 12358 raft_consensus.cc:2229] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:45:26.635262 12358 raft_consensus.cc:2258] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:45:26.635807 12358 ts_tablet_manager.cc:1905] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:45:26.638535 12358 ts_tablet_manager.cc:1918] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:45:26.638929 12358 log.cc:1196] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal/wals/d05a7503cc1948159c735d0c6cc81e10
I20260807 08:45:26.639254 12358 ts_tablet_manager.cc:1939] T d05a7503cc1948159c735d0c6cc81e10 P 75d2b3ec34fb4ca8b33e8a5476510841: Deleting consensus metadata
I20260807 08:45:26.639972 12123 catalog_manager.cc:4841] TS 75d2b3ec34fb4ca8b33e8a5476510841 (127.28.108.129:38665): tablet d05a7503cc1948159c735d0c6cc81e10 (table test-table [id=96beaff97dcf4f4697f3cfcb24d35067]) successfully deleted
W20260807 08:45:26.709017 12354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:26.709278 12354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:26.709306 12354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:26.710659 12354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:26.710716 12354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:26.710747 12354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:45:26.712385 12354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:46265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=34163
--tserver_master_addrs=127.28.108.190:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:26.712674 12354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:26.712965 12354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:26.719501 12367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:26.719611 12354 server_base.cc:1034] running on GCE node
W20260807 08:45:26.719487 12369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:26.719483 12365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:26.720233 12354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:26.720548 12354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:26.721737 12354 hybrid_clock.cc:648] HybridClock initialized: now 1786092326721725 us; error 36 us; skew 500 ppm
Aug 07 08:45:26 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:26.725356 12354 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:45:26.725942 12354 webserver.cc:460] Webserver started at http://127.28.108.130:34163/ using document root <none> and password file <none>
I20260807 08:45:26.726234 12354 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:26.726344 12354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:26.728085 12354 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:45:26.729183 12376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:26.729681 12354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:26.729807 12354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "b20980151bed4bc0a3b6a1139fe4d03c"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "1f583d6e956b6d379624cf4310fc44f7"
server_key_iv: "e0bc4f115d491f0899df1a457191d9b5"
server_key_version: "encryptionkey@0"
I20260807 08:45:26.730260 12354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:26.747478 12354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:26.750785 12354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:26.751020 12354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:26.751331 12354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:26.751729 12354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:26.751799 12354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:26.751852 12354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:26.751886 12354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:26.760334 12354 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:46265
I20260807 08:45:26.760380 12501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:46265 every 8 connection(s)
I20260807 08:45:26.760843 12354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:45:26.765051 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 12354
I20260807 08:45:26.765403 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:42437
--local_ip_for_outbound_sockets=127.28.108.131
--tserver_master_addrs=127.28.108.190:46821
--webserver_port=38333
--webserver_interface=127.28.108.131
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 08:45:26 dist-test-slave-9zdj krb5kdc[11504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:26.774156 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.762024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:60917 (local address 127.28.108.190:46821)
0807 08:45:26.762108 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.762110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.762582 (+ 472us) server_negotiation.cc:407] Connection header received
0807 08:45:26.763157 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.763159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.763173 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.763267 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:26.764724 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.765419 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.766696 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.766934 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.767018 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.770079 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:26.770081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:26.770081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:26.770090 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:26.772333 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:26.772907 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:26.772909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:26.772910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:26.772988 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:26.773324 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:26.773325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:26.773325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:26.773470 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:26.773554 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.773940 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.774053 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 08:45:26.774399 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.761913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:60917)
0807 08:45:26.762338 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.762418 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:26.763024 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:26.763433 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:26.763437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:26.764353 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:26.764591 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.764598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.765554 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.765556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.766573 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.766577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.767033 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.767035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.767183 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.767816 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:26.767825 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:26.769920 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:26.772475 (+ 2555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:26.772490 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:26.772493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:26.772806 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:26.773080 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:26.773081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:26.773081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:26.773242 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:26.773543 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:26.773551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:26.773780 (+ 229us) client_negotiation.cc:769] Sending connection context
0807 08:45:26.773916 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 08:45:26.774123 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260807 08:45:26.775023 12502 heartbeater.cc:347] Connected to a master server at 127.28.108.190:46821
I20260807 08:45:26.775170 12502 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:26.775421 12502 heartbeater.cc:511] Master 127.28.108.190:46821 requested a full tablet report, sending...
I20260807 08:45:26.775934 12141 ts_manager.cc:194] Registered new tserver with Master: b20980151bed4bc0a3b6a1139fe4d03c (127.28.108.130:46265)
I20260807 08:45:26.776523 12141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:60917
W20260807 08:45:26.861845 12505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:26.862113 12505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:26.862169 12505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:26.863809 12505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:26.863888 12505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:26.863940 12505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:45:26.865835 12505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:42437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=38333
--tserver_master_addrs=127.28.108.190:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:26.866169 12505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:26.866443 12505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:26.873056 12516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:26.873056 12512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:26.873059 12513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:26.873581 12505 server_base.cc:1034] running on GCE node
I20260807 08:45:26.873859 12505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:26.874171 12505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:26.875383 12505 hybrid_clock.cc:648] HybridClock initialized: now 1786092326875366 us; error 37 us; skew 500 ppm
Aug 07 08:45:26 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:26.879029 12505 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:45:26.879834 12505 webserver.cc:460] Webserver started at http://127.28.108.131:38333/ using document root <none> and password file <none>
I20260807 08:45:26.880167 12505 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:26.880283 12505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:26.882148 12505 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:45:26.883162 12524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:26.883451 12505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:45:26.883581 12505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "7081c41bf91d45dd9f00efbb5f9e954e"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "05be02bf9437887949e320ab2ea6dc44"
server_key_iv: "a1a3ed95a839b3b34c5cb071c8c5b9d9"
server_key_version: "encryptionkey@0"
I20260807 08:45:26.884030 12505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:26.907792 12505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:26.910890 12505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:26.911120 12505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:26.911427 12505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:26.911816 12505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:26.911885 12505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:26.911934 12505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:26.911972 12505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:26.919611 12505 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:42437
I20260807 08:45:26.919665 12646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:42437 every 8 connection(s)
I20260807 08:45:26.920217 12505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:45:26.923929 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 12505
Aug 07 08:45:26 dist-test-slave-9zdj krb5kdc[11504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:26.933329 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.924468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.28.108.190:46821)
0807 08:45:26.924605 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.924607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.924839 (+ 232us) server_negotiation.cc:407] Connection header received
0807 08:45:26.924922 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.924923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.924937 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.925042 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:26.926114 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.926659 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.927609 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.927928 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.928019 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.929771 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:26.929772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:26.929773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:26.929780 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:26.931758 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:26.932173 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:26.932174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:26.932175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:26.932373 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:26.932653 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:26.932654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:26.932655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:26.932806 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:26.933048 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.933090 (+ 42us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.933188 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260807 08:45:26.933475 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.921192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:54205 (local address 127.28.108.190:46821)
0807 08:45:26.921275 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.921277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.921541 (+ 264us) server_negotiation.cc:407] Connection header received
0807 08:45:26.922221 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.922223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.922238 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.922324 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:26.924129 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.924920 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.926540 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.926781 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.926887 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.929913 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:26.929914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:26.929915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:26.929923 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:26.931663 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:26.932263 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:26.932264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:26.932265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:26.932387 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:26.932713 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:26.932714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:26.932714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:26.932858 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:26.932955 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.933287 (+ 332us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.933385 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 08:45:26.933882 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.921178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:54205)
0807 08:45:26.921382 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.921432 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:26.922080 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:26.922436 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:26.922440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:26.923202 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:26.923909 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.923918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.925035 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.925036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.926427 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.926430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.926906 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.926907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.927090 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.927756 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:26.927768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:26.929798 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:26.931782 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:26.931792 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:26.931795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:26.932174 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:26.932471 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:26.932471 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:26.932472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:26.932628 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:26.932948 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:26.932956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:26.933129 (+ 173us) client_negotiation.cc:769] Sending connection context
0807 08:45:26.933245 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 08:45:26.933436 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":120,"threads_started":1}
I20260807 08:45:26.934382 12647 heartbeater.cc:347] Connected to a master server at 127.28.108.190:46821
I20260807 08:45:26.934499 12647 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:26.934720 12647 heartbeater.cc:511] Master 127.28.108.190:46821 requested a full tablet report, sending...
I20260807 08:45:26.935184 12139 ts_manager.cc:194] Registered new tserver with Master: 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437)
I20260807 08:45:26.935631 12139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:54205
I20260807 08:45:26.946030 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:45:26.949400 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.946524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.28.108.190:46821)
0807 08:45:26.946604 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.946606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.946652 (+ 46us) server_negotiation.cc:407] Connection header received
0807 08:45:26.946761 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.946762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.946804 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.946874 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:45:26.947082 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.947594 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.948360 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.948568 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.948626 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.948886 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:45:26.949007 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:45:26.949088 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.949280 (+ 192us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.949318 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 08:45:26.949762 12139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:45:26.951493 12139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:45:26.954651 12275 tablet_service.cc:1467] Processing CreateTablet for tablet e5ecd948debd40ecbd51c11b4c5de7e5 (DEFAULT_TABLE table=test-table [id=3188fa2eea604df4add4fedc7fde1231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:26.954923 12275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ecd948debd40ecbd51c11b4c5de7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:26.956868 12231 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: Bootstrap starting.
I20260807 08:45:26.957291 12231 tablet_bootstrap.cc:654] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:26.958264 12231 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: No bootstrap required, opened a new log
I20260807 08:45:26.958336 12231 ts_tablet_manager.cc:1397] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260807 08:45:26.958752 12231 raft_consensus.cc:348] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:26.958855 12231 raft_consensus.cc:374] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:26.958917 12231 raft_consensus.cc:729] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d2b3ec34fb4ca8b33e8a5476510841, State: Initialized, Role: FOLLOWER
I20260807 08:45:26.958974 12231 consensus_queue.cc:260] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:26.959033 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.954036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:42437 (local address 127.0.0.1:55542)
0807 08:45:26.954106 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.954119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:26.954166 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:26.954763 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:26.954764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:26.954766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:26.954827 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.954829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.956147 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.956148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.957459 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.957463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.958644 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.958645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.958740 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.958776 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 08:45:26.958879 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 08:45:26.958942 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 08:45:26.959045 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.954174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.28.108.131:42437)
0807 08:45:26.954546 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.954550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.954562 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:45:26.954600 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.954602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.954621 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.954724 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:26.954903 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.956009 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.957578 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.958514 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.958628 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.958708 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.958903 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.958953 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":103,"threads_started":1}
I20260807 08:45:26.959105 12231 ts_tablet_manager.cc:1428] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260807 08:45:26.959456 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.954274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.28.108.130:46265)
0807 08:45:26.954525 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:26.954532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:26.954546 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:45:26.954598 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:26.954601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:26.954621 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:26.954738 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:26.955257 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.956478 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.957956 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.958868 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.958931 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.958993 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:26.959295 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 08:45:26.959349 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":117,"threads_started":1}
I20260807 08:45:26.959699 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:26.954153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:46265 (local address 127.0.0.1:59712)
0807 08:45:26.954314 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:26.954327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:26.954385 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:26.954929 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:26.954930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:26.954933 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:26.954972 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.954973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.956628 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.956629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.957835 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:26.957838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:26.959073 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:26.959074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:26.959156 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:26.959161 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:45:26.959542 (+ 381us) client_negotiation.cc:240] Negotiation successful
0807 08:45:26.959598 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20260807 08:45:26.959939 12573 tablet_service.cc:1467] Processing CreateTablet for tablet e5ecd948debd40ecbd51c11b4c5de7e5 (DEFAULT_TABLE table=test-table [id=3188fa2eea604df4add4fedc7fde1231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:26.960372 12573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ecd948debd40ecbd51c11b4c5de7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:26.960527 12429 tablet_service.cc:1467] Processing CreateTablet for tablet e5ecd948debd40ecbd51c11b4c5de7e5 (DEFAULT_TABLE table=test-table [id=3188fa2eea604df4add4fedc7fde1231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:26.960896 12429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ecd948debd40ecbd51c11b4c5de7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:26.965054 12659 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Bootstrap starting.
I20260807 08:45:26.965993 12659 tablet_bootstrap.cc:654] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:26.966027 12660 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Bootstrap starting.
I20260807 08:45:26.966648 12659 log.cc:824] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:26.966890 12660 tablet_bootstrap.cc:654] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:26.967489 12660 log.cc:824] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:26.967643 12659 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: No bootstrap required, opened a new log
I20260807 08:45:26.967737 12659 ts_tablet_manager.cc:1397] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:45:26.968439 12660 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: No bootstrap required, opened a new log
I20260807 08:45:26.968530 12660 ts_tablet_manager.cc:1397] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:45:26.970039 12659 raft_consensus.cc:348] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:26.970274 12659 raft_consensus.cc:374] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:26.970332 12659 raft_consensus.cc:729] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20980151bed4bc0a3b6a1139fe4d03c, State: Initialized, Role: FOLLOWER
I20260807 08:45:26.970475 12659 consensus_queue.cc:260] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:26.970706 12660 raft_consensus.cc:348] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:26.970836 12659 ts_tablet_manager.cc:1428] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:45:26.970853 12502 heartbeater.cc:503] Master 127.28.108.190:46821 was elected leader, sending a full tablet report...
I20260807 08:45:26.970876 12660 raft_consensus.cc:374] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:26.970918 12660 raft_consensus.cc:729] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7081c41bf91d45dd9f00efbb5f9e954e, State: Initialized, Role: FOLLOWER
I20260807 08:45:26.971028 12660 consensus_queue.cc:260] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:26.971379 12660 ts_tablet_manager.cc:1428] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:45:26.971475 12647 heartbeater.cc:503] Master 127.28.108.190:46821 was elected leader, sending a full tablet report...
W20260807 08:45:27.011593 12503 tablet.cc:2369] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:45:27.070945 12355 raft_consensus.cc:482] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:45:27.071194 12355 raft_consensus.cc:504] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:27.071555 12355 leader_election.cc:288] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437), b20980151bed4bc0a3b6a1139fe4d03c (127.28.108.130:46265)
I20260807 08:45:27.075012 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.071989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:41111 (local address 127.28.108.130:46265)
0807 08:45:27.072039 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:27.072041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:27.072163 (+ 122us) server_negotiation.cc:407] Connection header received
0807 08:45:27.072254 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:27.072255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:27.072268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:27.072332 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:27.072504 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:27.073009 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:27.074159 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:27.074573 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:27.074665 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:27.074677 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:27.074886 (+ 209us) server_negotiation.cc:299] Negotiation successful
0807 08:45:27.074933 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 08:45:27.075048 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.071936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:45767 (local address 127.28.108.131:42437)
0807 08:45:27.072023 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:27.072025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:27.072036 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:45:27.072069 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:27.072070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:27.072079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:27.072143 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:27.072356 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:27.072909 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:27.074174 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:27.074622 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:27.074735 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:27.074746 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:27.074919 (+ 173us) server_negotiation.cc:299] Negotiation successful
0807 08:45:27.074967 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 08:45:27.075062 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.071801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:42437 (local address 127.28.108.129:45767)
0807 08:45:27.071876 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.071890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:27.071957 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:27.072158 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:27.072159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:27.072161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:27.072255 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:27.072257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:27.073019 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:27.073020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:27.074061 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:27.074064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:27.074735 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:27.074736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:27.074830 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:27.074833 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:45:27.074911 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 08:45:27.074963 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 08:45:27.075381 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.071816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:46265 (local address 127.28.108.129:41111)
0807 08:45:27.072075 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.072084 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:27.072182 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:27.072334 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:27.072335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:27.072337 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:27.072423 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:27.072425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:27.073109 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:27.073110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:27.074048 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:27.074052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:27.074692 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:27.074693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:27.074766 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:27.074792 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 08:45:27.074875 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 08:45:27.074921 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":3,"thread_start_us":120,"threads_started":1}
I20260807 08:45:27.075489 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ecd948debd40ecbd51c11b4c5de7e5" candidate_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" is_pre_election: true
I20260807 08:45:27.075700 12596 raft_consensus.cc:2454] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75d2b3ec34fb4ca8b33e8a5476510841 in term 0.
I20260807 08:45:27.075846 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ecd948debd40ecbd51c11b4c5de7e5" candidate_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" is_pre_election: true
I20260807 08:45:27.076059 12450 raft_consensus.cc:2454] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75d2b3ec34fb4ca8b33e8a5476510841 in term 0.
I20260807 08:45:27.076056 12227 leader_election.cc:302] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7081c41bf91d45dd9f00efbb5f9e954e, 75d2b3ec34fb4ca8b33e8a5476510841; no voters:
I20260807 08:45:27.076304 12355 raft_consensus.cc:2781] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:45:27.076375 12355 raft_consensus.cc:482] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:45:27.076399 12355 raft_consensus.cc:3037] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:27.077216 12355 raft_consensus.cc:504] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:27.077421 12355 leader_election.cc:288] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [CANDIDATE]: Term 1 election: Requested vote from peers 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437), b20980151bed4bc0a3b6a1139fe4d03c (127.28.108.130:46265)
I20260807 08:45:27.077572 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ecd948debd40ecbd51c11b4c5de7e5" candidate_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7081c41bf91d45dd9f00efbb5f9e954e"
I20260807 08:45:27.077724 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ecd948debd40ecbd51c11b4c5de7e5" candidate_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20980151bed4bc0a3b6a1139fe4d03c"
I20260807 08:45:27.077752 12596 raft_consensus.cc:3037] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:27.077812 12450 raft_consensus.cc:3037] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:27.078787 12450 raft_consensus.cc:2454] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75d2b3ec34fb4ca8b33e8a5476510841 in term 1.
I20260807 08:45:27.078787 12596 raft_consensus.cc:2454] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75d2b3ec34fb4ca8b33e8a5476510841 in term 1.
I20260807 08:45:27.079090 12227 leader_election.cc:302] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7081c41bf91d45dd9f00efbb5f9e954e, 75d2b3ec34fb4ca8b33e8a5476510841; no voters:
I20260807 08:45:27.079223 12355 raft_consensus.cc:2781] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:45:27.079553 12355 raft_consensus.cc:686] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 LEADER]: Becoming Leader. State: Replica: 75d2b3ec34fb4ca8b33e8a5476510841, State: Running, Role: LEADER
I20260807 08:45:27.079643 12355 consensus_queue.cc:237] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:27.080654 12141 catalog_manager.cc:5495] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75d2b3ec34fb4ca8b33e8a5476510841 (127.28.108.129). New cstate: current_term: 1 leader_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } health_report { overall_health: HEALTHY } } }
W20260807 08:45:27.104288 12350 tablet.cc:2369] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:45:27.111639 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.108128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.28.108.129:38665)
0807 08:45:27.108389 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:27.108392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:27.108404 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:45:27.108450 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:27.108451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:27.108473 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:27.108565 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:45:27.108734 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:27.109432 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:27.110602 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:27.110879 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:27.110949 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:27.111168 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:45:27.111365 (+ 197us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:45:27.111441 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:27.111537 (+ 96us) server_negotiation.cc:299] Negotiation successful
0807 08:45:27.111564 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260807 08:45:27.113889 12596 raft_consensus.cc:1261] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 1 FOLLOWER]: Refusing update from remote peer 75d2b3ec34fb4ca8b33e8a5476510841: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:45:27.113879 12450 raft_consensus.cc:1261] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Refusing update from remote peer 75d2b3ec34fb4ca8b33e8a5476510841: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:45:27.114377 12355 consensus_queue.cc:1035] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:45:27.114627 12666 consensus_queue.cc:1035] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:45:27.118465 12671 mvcc.cc:205] Tried to move back new op lower bound from 7315834171854548992 to 7315834171719208960. Current Snapshot: MvccSnapshot[applied={T|T < 7315834171854548992}]
I20260807 08:45:27.119134 12672 mvcc.cc:205] Tried to move back new op lower bound from 7315834171854548992 to 7315834171719208960. Current Snapshot: MvccSnapshot[applied={T|T < 7315834171854548992 or (T in {7315834171854548992})}]
I20260807 08:45:27.120546 12141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:45:27.120677 12141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:45:27.121687 12141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Sending DeleteTablet for 3 replicas of tablet e5ecd948debd40ecbd51c11b4c5de7e5
I20260807 08:45:27.122141 12573 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ecd948debd40ecbd51c11b4c5de7e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:27 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:55542
I20260807 08:45:27.122177 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 12183
I20260807 08:45:27.122370 12275 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ecd948debd40ecbd51c11b4c5de7e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:27 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:35064
I20260807 08:45:27.122475 12429 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ecd948debd40ecbd51c11b4c5de7e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:27 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:59712
I20260807 08:45:27.126964 12680 tablet_replica.cc:330] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: stopping tablet replica
I20260807 08:45:27.127167 12680 raft_consensus.cc:2229] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:45:27.127341 12680 raft_consensus.cc:2258] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:45:27.127887 12680 ts_tablet_manager.cc:1905] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:45:27.129839 12680 ts_tablet_manager.cc:1918] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:45:27.129957 12680 log.cc:1196] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/e5ecd948debd40ecbd51c11b4c5de7e5
I20260807 08:45:27.130317 12680 ts_tablet_manager.cc:1939] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Deleting consensus metadata
I20260807 08:45:27.131022 12125 catalog_manager.cc:4841] TS 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437): tablet e5ecd948debd40ecbd51c11b4c5de7e5 (table test-table [id=3188fa2eea604df4add4fedc7fde1231]) successfully deleted
W20260807 08:45:27.132102 12123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:38665 (error 108)
I20260807 08:45:27.132409 12681 tablet_replica.cc:330] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: stopping tablet replica
I20260807 08:45:27.132550 12681 raft_consensus.cc:2229] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:45:27.132709 12681 raft_consensus.cc:2258] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:45:27.132997 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.132845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:38665 (local address 127.0.0.1:35082)
0807 08:45:27.132912 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.132965 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:38665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 08:45:27.133249 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 12354
W20260807 08:45:27.133195 12123 catalog_manager.cc:4584] TS 75d2b3ec34fb4ca8b33e8a5476510841 (127.28.108.129:38665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5ecd948debd40ecbd51c11b4c5de7e5: Network error: Client connection negotiation failed: client connection to 127.28.108.129:38665: connect: Connection refused (error 111)
I20260807 08:45:27.141008 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 12505
I20260807 08:45:27.141109 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.140962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:46265 (local address 127.0.0.1:59720)
0807 08:45:27.141033 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.141070 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.130:46265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 08:45:27.148947 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 12105
I20260807 08:45:27.158440 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:46821
--webserver_interface=127.28.108.190
--webserver_port=45011
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:27.250694 12684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:27.251003 12684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:27.251049 12684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:27.252395 12684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:45:27.252483 12684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:27.252507 12684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:27.252530 12684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:45:27.252563 12684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:45:27.254356 12684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:46821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=45011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:27.254694 12684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:27.254956 12684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:27.260771 12693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:27.260810 12691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:27.260771 12690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:27.260998 12684 server_base.cc:1034] running on GCE node
I20260807 08:45:27.261577 12684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:27.262115 12684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:27.263306 12684 hybrid_clock.cc:648] HybridClock initialized: now 1786092327263289 us; error 31 us; skew 500 ppm
Aug 07 08:45:27 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:27.267241 12684 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:45:27.268004 12684 webserver.cc:460] Webserver started at http://127.28.108.190:45011/ using document root <none> and password file <none>
I20260807 08:45:27.268293 12684 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:27.268375 12684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:27.270197 12684 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:45:27.271212 12702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:27.271517 12684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:45:27.271737 12684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "3646fe2eb3bb41918de1e475d719be14"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "3a52b7f9d07bb82ab22f9ac08efaeede"
server_key_iv: "87622244fa7ebb9bec758c616e269e6a"
server_key_version: "encryptionkey@0"
I20260807 08:45:27.272168 12684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:27.301381 12684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:27.304394 12684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:27.304616 12684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:27.310079 12684 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:46821
I20260807 08:45:27.310141 12765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:46821 every 8 connection(s)
I20260807 08:45:27.310603 12684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:45:27.312512 12766 sys_catalog.cc:263] Verifying existing consensus state
I20260807 08:45:27.313416 12766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Bootstrap starting.
I20260807 08:45:27.316646 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 12684
I20260807 08:45:27.317035 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:38665
--local_ip_for_outbound_sockets=127.28.108.129
--tserver_master_addrs=127.28.108.190:46821
--webserver_port=36851
--webserver_interface=127.28.108.129
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:45:27.324195 12766 log.cc:824] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:27.327384 12766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:45:27.327873 12766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14: Bootstrap complete.
I20260807 08:45:27.330742 12766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:27.331028 12766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3646fe2eb3bb41918de1e475d719be14, State: Initialized, Role: FOLLOWER
I20260807 08:45:27.331223 12766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:27.331316 12766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:45:27.331389 12766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:45:27.331445 12766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:45:27.332859 12766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:27.332978 12766 leader_election.cc:302] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3646fe2eb3bb41918de1e475d719be14; no voters:
I20260807 08:45:27.333215 12766 leader_election.cc:288] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 08:45:27.333338 12771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:45:27.333561 12766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:45:27.333572 12771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [term 3 LEADER]: Becoming Leader. State: Replica: 3646fe2eb3bb41918de1e475d719be14, State: Running, Role: LEADER
I20260807 08:45:27.333876 12771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } }
I20260807 08:45:27.334229 12772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3646fe2eb3bb41918de1e475d719be14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } } }
I20260807 08:45:27.334321 12772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:27.334510 12775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3646fe2eb3bb41918de1e475d719be14. Latest consensus state: current_term: 3 leader_uuid: "3646fe2eb3bb41918de1e475d719be14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3646fe2eb3bb41918de1e475d719be14" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46821 } } }
I20260807 08:45:27.334614 12775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3646fe2eb3bb41918de1e475d719be14 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:27.334748 12783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:45:27.335683 12783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:45:27.335803 12783 catalog_manager.cc:1259] Loaded cluster ID: 8e14d6eb684d446b8b8904ebb163868a
I20260807 08:45:27.335842 12783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 08:45:27.336566 12783 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f89e06029ad google::LogMessage::Fail() at ??:0
@ 0x7f89e060588a google::LogMessage::SendToLog() at ??:0
@ 0x7f89e06024de google::LogMessage::Flush() at ??:0
@ 0x7f89e0603a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f89e85230f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f89e852c8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f89e24d00d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f89e24c5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f89e33296db start_thread at ??:0
@ 0x7f89df10061f clone at ??:0
W20260807 08:45:27.440047 12769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:27.440308 12769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:27.440338 12769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:27.441816 12769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:27.441881 12769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:27.441915 12769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:45:27.443775 12769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:38665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=36851
--tserver_master_addrs=127.28.108.190:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:27.444167 12769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:27.444500 12769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:27.450513 12811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:27.450523 12810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:27.450636 12813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:27.451097 12769 server_base.cc:1034] running on GCE node
I20260807 08:45:27.451391 12769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:27.451732 12769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:27.452931 12769 hybrid_clock.cc:648] HybridClock initialized: now 1786092327452906 us; error 46 us; skew 500 ppm
Aug 07 08:45:27 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:27.456820 12769 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:45:27.457429 12769 webserver.cc:460] Webserver started at http://127.28.108.129:36851/ using document root <none> and password file <none>
I20260807 08:45:27.457787 12769 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:27.457865 12769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:27.459396 12769 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:27.460264 12822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:27.460561 12769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:27.460690 12769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "75d2b3ec34fb4ca8b33e8a5476510841"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "6a242e7eeaa41effea06aa1ecdb5959d"
server_key_iv: "b5128c241312f374288c9f351bf23f21"
server_key_version: "encryptionkey@0"
I20260807 08:45:27.461158 12769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:27.479377 12769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:27.482429 12769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:27.482641 12769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:27.482935 12769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:27.483469 12832 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 08:45:27.486125 12769 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 08:45:27.486299 12769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260807 08:45:27.486366 12769 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 08:45:27.487406 12769 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 08:45:27.487479 12769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260807 08:45:27.487579 12832 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: Bootstrap starting.
I20260807 08:45:27.496256 12769 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:38665
I20260807 08:45:27.496387 12958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:38665 every 8 connection(s)
I20260807 08:45:27.496832 12769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:45:27.500732 12832 log.cc:824] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:27.504410 12832 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:45:27.504866 12832 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: Bootstrap complete.
I20260807 08:45:27.505796 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 12769
I20260807 08:45:27.506139 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:46265
--local_ip_for_outbound_sockets=127.28.108.130
--tserver_master_addrs=127.28.108.190:46821
--webserver_port=34163
--webserver_interface=127.28.108.130
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:45:27.506434 12832 ts_tablet_manager.cc:1397] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.012s
I20260807 08:45:27.508338 12832 raft_consensus.cc:348] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:27.508765 12832 raft_consensus.cc:729] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d2b3ec34fb4ca8b33e8a5476510841, State: Initialized, Role: FOLLOWER
I20260807 08:45:27.508951 12832 consensus_queue.cc:260] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:27.509353 12832 ts_tablet_manager.cc:1428] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
W20260807 08:45:27.596791 12961 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:45:27.497806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:47293)
0807 08:45:27.498019 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.498072 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:27.498617 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:27.596550 (+ 97933us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":127,"threads_started":1}
W20260807 08:45:27.597146 12959 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:46821 (0 consecutive failures): Network error: Failed to ping master at 127.28.108.190:46821: Client connection negotiation failed: client connection to 127.28.108.190:46821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 08:45:27.597594 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.597463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:34975)
0807 08:45:27.597509 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.597564 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260807 08:45:27.598007 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.597906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:41547)
0807 08:45:27.597957 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.597987 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260807 08:45:27.598206 12959 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 08:45:27.632037 12963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:27.632305 12963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:27.632334 12963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:27.633736 12963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:27.633797 12963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:27.633831 12963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:45:27.635514 12963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:46265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=34163
--tserver_master_addrs=127.28.108.190:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:27.635835 12963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:27.636102 12963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:27.642222 12971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:27.642231 12963 server_base.cc:1034] running on GCE node
W20260807 08:45:27.642273 12975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:27.642347 12973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:27.642755 12963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:27.643096 12963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:27.644402 12963 hybrid_clock.cc:648] HybridClock initialized: now 1786092327644278 us; error 42 us; skew 500 ppm
Aug 07 08:45:27 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:27.647671 12963 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:45:27.648314 12963 webserver.cc:460] Webserver started at http://127.28.108.130:34163/ using document root <none> and password file <none>
I20260807 08:45:27.648627 12963 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:27.648710 12963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:27.650712 12963 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:27.651685 12982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:27.651916 12963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:27.652067 12963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "b20980151bed4bc0a3b6a1139fe4d03c"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "1f583d6e956b6d379624cf4310fc44f7"
server_key_iv: "e0bc4f115d491f0899df1a457191d9b5"
server_key_version: "encryptionkey@0"
I20260807 08:45:27.652591 12963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:27.676908 12963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:27.679827 12963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:27.680054 12963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:27.680371 12963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:27.680935 12990 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 08:45:27.682744 12963 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 08:45:27.682861 12963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260807 08:45:27.682915 12963 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 08:45:27.683794 12963 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 08:45:27.683859 12963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260807 08:45:27.683990 12990 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Bootstrap starting.
I20260807 08:45:27.686244 12990 log.cc:824] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:27.689155 12990 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:45:27.689564 12990 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Bootstrap complete.
I20260807 08:45:27.689828 12990 ts_tablet_manager.cc:1397] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260807 08:45:27.691453 12990 raft_consensus.cc:348] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:27.691919 12990 raft_consensus.cc:729] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20980151bed4bc0a3b6a1139fe4d03c, State: Initialized, Role: FOLLOWER
I20260807 08:45:27.692404 12963 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:46265
I20260807 08:45:27.692339 12990 consensus_queue.cc:260] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:27.692490 13111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:46265 every 8 connection(s)
I20260807 08:45:27.692729 12990 ts_tablet_manager.cc:1428] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260807 08:45:27.692858 12963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:45:27.694198 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.693817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:60687)
0807 08:45:27.693995 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.694070 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
W20260807 08:45:27.694527 13115 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:46821 (0 consecutive failures): Network error: Failed to ping master at 127.28.108.190:46821: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
I20260807 08:45:27.695008 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.694898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:45785)
0807 08:45:27.694950 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.694985 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260807 08:45:27.695760 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.695580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:48615)
0807 08:45:27.695650 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.695735 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 08:45:27.695991 13115 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 08:45:27.696072 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 12963
I20260807 08:45:27.696429 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:42437
--local_ip_for_outbound_sockets=127.28.108.131
--tserver_master_addrs=127.28.108.190:46821
--webserver_port=38333
--webserver_interface=127.28.108.131
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:41237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:27.795907 13118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:27.796237 13118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:27.796291 13118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:27.797773 13118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:27.797873 13118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:27.797927 13118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:45:27.799825 13118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:41237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:42437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=38333
--tserver_master_addrs=127.28.108.190:46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:27.800197 13118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:27.800509 13118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:27.807000 13123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:27.807000 13124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:27.807009 13126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:27.807868 13118 server_base.cc:1034] running on GCE node
I20260807 08:45:27.808188 13118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:27.808517 13118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:27.809713 13118 hybrid_clock.cc:648] HybridClock initialized: now 1786092327809689 us; error 37 us; skew 500 ppm
Aug 07 08:45:27 dist-test-slave-9zdj krb5kdc[11504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:27.813360 13118 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:45:27.814065 13118 webserver.cc:460] Webserver started at http://127.28.108.131:38333/ using document root <none> and password file <none>
I20260807 08:45:27.814359 13118 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:27.814417 13118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:27.816078 13118 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:45:27.817008 13134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:27.817291 13118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:45:27.817423 13118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "7081c41bf91d45dd9f00efbb5f9e954e"
format_stamp: "Formatted at 2026-08-07 08:45:25 on dist-test-slave-9zdj"
server_key: "05be02bf9437887949e320ab2ea6dc44"
server_key_iv: "a1a3ed95a839b3b34c5cb071c8c5b9d9"
server_key_version: "encryptionkey@0"
I20260807 08:45:27.817987 13118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:27.830250 13118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:27.833519 13118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:27.833777 13118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:27.834156 13118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:27.834534 13118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:27.834595 13118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:27.834643 13118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:27.834671 13118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:27.842185 13118 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:42437
I20260807 08:45:27.842226 13262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:42437 every 8 connection(s)
I20260807 08:45:27.842651 13118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:45:27.844034 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.843546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:39929)
0807 08:45:27.843786 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.843877 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":150,"threads_started":1}
I20260807 08:45:27.844403 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 13118
W20260807 08:45:27.844609 13263 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:46821 (0 consecutive failures): Network error: Failed to ping master at 127.28.108.190:46821: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
I20260807 08:45:27.845794 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.845641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:51011)
0807 08:45:27.845706 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.845748 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260807 08:45:27.846282 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:27.846064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:36355)
0807 08:45:27.846121 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:27.846259 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 08:45:27.846448 13263 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 08:45:28.599125 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:28.598802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:47443)
0807 08:45:28.599006 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:28.599068 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260807 08:45:28.696833 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:28.696515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:37023)
0807 08:45:28.696677 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:28.696784 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260807 08:45:28.847328 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:28.846916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:50165)
0807 08:45:28.847096 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:28.847157 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260807 08:45:29.124873 13270 raft_consensus.cc:482] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:45:29.125058 13270 raft_consensus.cc:504] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:29.125588 13270 leader_election.cc:288] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437), 75d2b3ec34fb4ca8b33e8a5476510841 (127.28.108.129:38665)
Aug 07 08:45:29 dist-test-slave-9zdj krb5kdc[11504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.129@KRBTEST.COM
Aug 07 08:45:29 dist-test-slave-9zdj krb5kdc[11504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.131@KRBTEST.COM
I20260807 08:45:29.139889 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:29.125899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:38665 (local address 127.28.108.130:49357)
0807 08:45:29.126041 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:29.126053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:29.126214 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:29.126550 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:29.126555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:29.127017 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:29.127322 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:29.127329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:29.128909 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:29.128910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:29.130044 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:29.130047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:29.131080 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:29.131081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:29.131243 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:29.131955 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:29.131967 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:29.134365 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:29.137371 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:29.137372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:29.137373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:29.137962 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:29.138512 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:29.138514 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:29.138514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:29.138637 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:29.139335 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:29.139336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:29.139385 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 08:45:29.139495 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 08:45:29.139746 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260807 08:45:29.139930 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:29.126020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:49357 (local address 127.28.108.129:38665)
0807 08:45:29.126169 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:29.126175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:29.126193 (+ 18us) server_negotiation.cc:407] Connection header received
0807 08:45:29.126387 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:29.126389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:29.126409 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:29.126483 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:29.127481 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:29.128772 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:29.130147 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:29.130971 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:29.131103 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:29.134926 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:29.134932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:29.134932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:29.134945 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:29.137209 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:29.138088 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:29.138092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:29.138092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:29.138269 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:29.138820 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:29.138821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:29.138821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:29.139234 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:29.139338 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:29.139609 (+ 271us) server_negotiation.cc:299] Negotiation successful
0807 08:45:29.139806 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260807 08:45:29.139938 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:29.125978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:50729 (local address 127.28.108.131:42437)
0807 08:45:29.126235 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:29.126246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:29.126281 (+ 35us) server_negotiation.cc:407] Connection header received
0807 08:45:29.126893 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:29.126897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:29.126947 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:29.127101 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:29.128099 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:29.129394 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:29.130494 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:29.131010 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:29.131569 (+ 559us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:29.134926 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:29.134932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:29.134932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:29.134945 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:29.137177 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:29.137983 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:29.137988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:29.137988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:29.138136 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:29.138536 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:29.138537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:29.138538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:29.138981 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:29.139064 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:29.139591 (+ 527us) server_negotiation.cc:299] Negotiation successful
0807 08:45:29.139824 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":62,"threads_started":1}
I20260807 08:45:29.140604 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:29.125879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:42437 (local address 127.28.108.130:50729)
0807 08:45:29.125949 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:29.125981 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:29.126192 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:29.127120 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:29.127121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:29.127864 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:29.127937 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:29.127940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:29.129519 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:29.129520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:29.130383 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:29.130386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:29.131133 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:29.131134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:29.131339 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:29.131955 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:29.131967 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:29.134782 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:29.137313 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:29.137325 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:29.137329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:29.137859 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:29.138230 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:29.138231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:29.138231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:29.138414 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:29.139089 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:29.139098 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:29.139333 (+ 235us) client_negotiation.cc:769] Sending connection context
0807 08:45:29.139448 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 08:45:29.139842 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":165}
I20260807 08:45:29.140906 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ecd948debd40ecbd51c11b4c5de7e5" candidate_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" is_pre_election: true
I20260807 08:45:29.141111 12907 raft_consensus.cc:2454] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20980151bed4bc0a3b6a1139fe4d03c in term 1.
I20260807 08:45:29.141500 12983 leader_election.cc:302] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75d2b3ec34fb4ca8b33e8a5476510841, b20980151bed4bc0a3b6a1139fe4d03c; no voters:
I20260807 08:45:29.141695 13270 raft_consensus.cc:2781] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 08:45:29.141798 13270 raft_consensus.cc:482] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:45:29.141839 13270 raft_consensus.cc:3037] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 1 FOLLOWER]: Advancing to term 2
W20260807 08:45:29.141903 12985 leader_election.cc:341] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437): Not found: Tablet not found: e5ecd948debd40ecbd51c11b4c5de7e5
I20260807 08:45:29.143272 13270 raft_consensus.cc:504] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:29.143472 13270 leader_election.cc:288] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [CANDIDATE]: Term 2 election: Requested vote from peers 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437), 75d2b3ec34fb4ca8b33e8a5476510841 (127.28.108.129:38665)
I20260807 08:45:29.143720 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ecd948debd40ecbd51c11b4c5de7e5" candidate_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "75d2b3ec34fb4ca8b33e8a5476510841"
I20260807 08:45:29.143863 12907 raft_consensus.cc:3037] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 1 FOLLOWER]: Advancing to term 2
W20260807 08:45:29.143894 12985 leader_election.cc:341] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437): Not found: Tablet not found: e5ecd948debd40ecbd51c11b4c5de7e5
I20260807 08:45:29.145306 12907 raft_consensus.cc:2454] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20980151bed4bc0a3b6a1139fe4d03c in term 2.
I20260807 08:45:29.145550 12983 leader_election.cc:302] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d2b3ec34fb4ca8b33e8a5476510841, b20980151bed4bc0a3b6a1139fe4d03c; no voters: 7081c41bf91d45dd9f00efbb5f9e954e
I20260807 08:45:29.145709 13270 raft_consensus.cc:2781] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:45:29.145902 13270 raft_consensus.cc:686] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [term 2 LEADER]: Becoming Leader. State: Replica: b20980151bed4bc0a3b6a1139fe4d03c, State: Running, Role: LEADER
I20260807 08:45:29.146051 13270 consensus_queue.cc:237] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:29.146428 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:29.146214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:51675)
0807 08:45:29.146259 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:29.146396 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260807 08:45:29.600277 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:29.599954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:34501)
0807 08:45:29.600169 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:29.600223 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
W20260807 08:45:29.693786 12985 consensus_peers.cc:487] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c -> Peer 7081c41bf91d45dd9f00efbb5f9e954e (127.28.108.131:42437): Couldn't send request to peer 7081c41bf91d45dd9f00efbb5f9e954e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5ecd948debd40ecbd51c11b4c5de7e5. This is attempt 1: this message will repeat every 5th retry.
I20260807 08:45:29.737377 12907 raft_consensus.cc:1261] T e5ecd948debd40ecbd51c11b4c5de7e5 P 75d2b3ec34fb4ca8b33e8a5476510841 [term 2 FOLLOWER]: Refusing update from remote peer b20980151bed4bc0a3b6a1139fe4d03c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260807 08:45:29.737972 13277 consensus_queue.cc:1035] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 08:45:29.739142 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:29.738997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:51203)
0807 08:45:29.739063 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:29.739105 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260807 08:45:29.740203 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:29.739978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:54041)
0807 08:45:29.740113 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:29.740152 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260807 08:45:29.740309 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:29.740196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:50693)
0807 08:45:29.740238 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:29.740276 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260807 08:45:29.848297 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:29.847983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:52141)
0807 08:45:29.848193 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:29.848243 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260807 08:45:30.108546 13285 ts_tablet_manager.cc:927] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Initiating tablet copy from peer b20980151bed4bc0a3b6a1139fe4d03c (127.28.108.130:46265)
I20260807 08:45:30.109037 13285 tablet_copy_client.cc:316] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: tablet copy: Beginning tablet copy session from remote peer at address 127.28.108.130:46265
Aug 07 08:45:30 dist-test-slave-9zdj krb5kdc[11504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.130@KRBTEST.COM
I20260807 08:45:30.119942 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:30.109787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:46265 (local address 127.28.108.131:38683)
0807 08:45:30.109945 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:30.109970 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:30.110067 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:30.110288 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:30.110296 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:30.110643 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:30.110712 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:30.110715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:30.111719 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:30.111728 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:30.112365 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:30.112368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:30.112925 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:30.112926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:30.113032 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:30.113705 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:30.113711 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:30.115919 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:30.118459 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:30.118475 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:30.118483 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:30.118747 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:30.119022 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:30.119023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:30.119023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:30.119094 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:30.119478 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:30.119491 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:30.119565 (+ 74us) client_negotiation.cc:769] Sending connection context
0807 08:45:30.119663 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 08:45:30.119815 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260807 08:45:30.119949 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:30.109882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:38683 (local address 127.28.108.130:46265)
0807 08:45:30.110093 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:30.110101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:30.110114 (+ 13us) server_negotiation.cc:407] Connection header received
0807 08:45:30.110155 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:30.110157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:30.110170 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:30.110282 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:30.110847 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:30.111593 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:30.112477 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:30.112824 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:30.112929 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:30.116076 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:30.116092 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:30.116093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:30.116109 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:30.118318 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:30.118865 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:30.118874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:30.118875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:30.118943 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:30.119164 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:30.119165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:30.119166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:30.119372 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:30.119506 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:30.119703 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 08:45:30.119816 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":101,"threads_started":1}
I20260807 08:45:30.120568 13081 tablet_copy_service.cc:140] P b20980151bed4bc0a3b6a1139fe4d03c: Received BeginTabletCopySession request for tablet e5ecd948debd40ecbd51c11b4c5de7e5 from peer 7081c41bf91d45dd9f00efbb5f9e954e ({username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:38683)
I20260807 08:45:30.120644 13081 tablet_copy_service.cc:161] P b20980151bed4bc0a3b6a1139fe4d03c: Beginning new tablet copy session on tablet e5ecd948debd40ecbd51c11b4c5de7e5 from peer 7081c41bf91d45dd9f00efbb5f9e954e at {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:38683: session id = 7081c41bf91d45dd9f00efbb5f9e954e-e5ecd948debd40ecbd51c11b4c5de7e5
I20260807 08:45:30.121286 13081 tablet_copy_source_session.cc:205] T e5ecd948debd40ecbd51c11b4c5de7e5 P b20980151bed4bc0a3b6a1139fe4d03c: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 08:45:30.122897 13285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ecd948debd40ecbd51c11b4c5de7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:30.126379 13285 tablet_copy_client.cc:791] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: tablet copy: Starting download of 0 data blocks...
I20260807 08:45:30.126586 13285 tablet_copy_client.cc:655] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: tablet copy: Starting download of 1 WAL segments...
I20260807 08:45:30.128036 13285 tablet_copy_client.cc:531] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 08:45:30.129253 13285 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Bootstrap starting.
I20260807 08:45:30.130991 13285 log.cc:824] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:30.134424 13285 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:45:30.134786 13285 tablet_bootstrap.cc:492] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Bootstrap complete.
I20260807 08:45:30.134984 13285 ts_tablet_manager.cc:1397] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 08:45:30.136368 13285 raft_consensus.cc:348] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:30.136632 13285 raft_consensus.cc:729] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7081c41bf91d45dd9f00efbb5f9e954e, State: Initialized, Role: FOLLOWER
I20260807 08:45:30.136795 13285 consensus_queue.cc:260] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7081c41bf91d45dd9f00efbb5f9e954e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 42437 } } peers { permanent_uuid: "b20980151bed4bc0a3b6a1139fe4d03c" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 46265 } } peers { permanent_uuid: "75d2b3ec34fb4ca8b33e8a5476510841" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 38665 } }
I20260807 08:45:30.137202 13285 ts_tablet_manager.cc:1428] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 08:45:30.137709 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:30.137513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:36625)
0807 08:45:30.137581 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:30.137668 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 08:45:30.137790 13081 tablet_copy_service.cc:342] P b20980151bed4bc0a3b6a1139fe4d03c: Request end of tablet copy session 7081c41bf91d45dd9f00efbb5f9e954e-e5ecd948debd40ecbd51c11b4c5de7e5 received from {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:38683
I20260807 08:45:30.137876 13081 tablet_copy_service.cc:433] P b20980151bed4bc0a3b6a1139fe4d03c: ending tablet copy session 7081c41bf91d45dd9f00efbb5f9e954e-e5ecd948debd40ecbd51c11b4c5de7e5 on tablet e5ecd948debd40ecbd51c11b4c5de7e5 with peer 7081c41bf91d45dd9f00efbb5f9e954e
I20260807 08:45:30.138339 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:30.138078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:43851)
0807 08:45:30.138165 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:30.138219 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 08:45:30.513127 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:30.512893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:34231)
0807 08:45:30.513025 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:30.513078 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":90,"threads_started":1}
I20260807 08:45:30.513085 13211 raft_consensus.cc:1203] T e5ecd948debd40ecbd51c11b4c5de7e5 P 7081c41bf91d45dd9f00efbb5f9e954e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260807 08:45:30.513696 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:30.513514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:47905)
0807 08:45:30.513575 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:30.513610 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260807 08:45:30.741119 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:30.740860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:50585)
0807 08:45:30.741012 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:30.741054 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260807 08:45:31.514140 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:31.513800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:39479)
0807 08:45:31.513997 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:31.514070 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
I20260807 08:45:31.514673 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:31.514451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:54671)
0807 08:45:31.514594 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:31.514633 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260807 08:45:31.742285 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:31.741983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:40573)
0807 08:45:31.742181 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:31.742233 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
I20260807 08:45:32.515226 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:32.514920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:45089)
0807 08:45:32.515114 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:32.515160 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20260807 08:45:32.515456 13302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:32.515212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:35263)
0807 08:45:32.515355 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:32.515402 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260807 08:45:32.743327 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:32.743010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:55233)
0807 08:45:32.743160 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:32.743272 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260807 08:45:33.516242 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:33.516005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:53233)
0807 08:45:33.516148 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:33.516190 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":53,"threads_started":1}
I20260807 08:45:33.516386 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:33.515965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:33215)
0807 08:45:33.516151 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:33.516198 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260807 08:45:33.744447 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:33.743979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:51551)
0807 08:45:33.744329 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:33.744389 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":81,"threads_started":1}
I20260807 08:45:34.517355 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:34.517069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:39105)
0807 08:45:34.517251 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:34.517300 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":125,"threads_started":1}
I20260807 08:45:34.517390 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:34.516979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:60311)
0807 08:45:34.517297 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:34.517340 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":232,"threads_started":1}
I20260807 08:45:34.745764 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:34.745352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:57775)
0807 08:45:34.745545 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:34.745690 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260807 08:45:35.518445 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:35.518083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:35571)
0807 08:45:35.518288 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:35.518343 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":117,"threads_started":1}
I20260807 08:45:35.518554 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:35.518097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:37145)
0807 08:45:35.518408 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:35.518470 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":227,"threads_started":1}
I20260807 08:45:35.746829 13314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:35.746535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:34445)
0807 08:45:35.746724 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:35.746774 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
I20260807 08:45:36.519608 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:36.519174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:34379)
0807 08:45:36.519487 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:36.519540 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":90,"threads_started":1}
I20260807 08:45:36.519788 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:36.519174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:46181)
0807 08:45:36.519391 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:36.519722 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
I20260807 08:45:36.748010 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:36.747706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:49131)
0807 08:45:36.747901 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:36.747957 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260807 08:45:37.520552 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:37.520262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:60289)
0807 08:45:37.520459 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:37.520500 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":117,"threads_started":1}
I20260807 08:45:37.520598 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:37.520379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:47583)
0807 08:45:37.520513 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:37.520554 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260807 08:45:37.749037 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:37.748744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:41471)
0807 08:45:37.748926 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:37.748981 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260807 08:45:38.521577 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:38.521238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:43803)
0807 08:45:38.521432 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:38.521510 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":136,"threads_started":1}
I20260807 08:45:38.521605 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:38.521351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:52703)
0807 08:45:38.521521 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:38.521560 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260807 08:45:38.750164 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:38.749876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:58673)
0807 08:45:38.750063 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:38.750111 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260807 08:45:39.522596 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:39.522328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:44799)
0807 08:45:39.522493 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:39.522545 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260807 08:45:39.522596 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:39.522327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:39723)
0807 08:45:39.522515 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:39.522548 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":105,"threads_started":1}
I20260807 08:45:39.751230 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:39.750932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:60953)
0807 08:45:39.751114 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:39.751169 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260807 08:45:40.523572 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:40.523224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:51317)
0807 08:45:40.523401 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:40.523454 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260807 08:45:40.523633 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:40.523391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:39457)
0807 08:45:40.523554 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:40.523589 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260807 08:45:40.752180 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:40.751900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:34125)
0807 08:45:40.752076 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:40.752124 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
I20260807 08:45:41.524597 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:41.524349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:45157)
0807 08:45:41.524516 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:41.524553 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260807 08:45:41.524600 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:41.524271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:41363)
0807 08:45:41.524455 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:41.524517 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":122,"threads_started":1}
I20260807 08:45:41.753307 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:41.753031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:34113)
0807 08:45:41.753178 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:41.753232 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260807 08:45:42.525532 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:42.525262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.131:45939)
0807 08:45:42.525433 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:42.525480 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260807 08:45:42.525552 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:42.525299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.130:37005)
0807 08:45:42.525455 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:42.525504 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260807 08:45:42.754343 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:42.754034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:46821 (local address 127.28.108.129:38095)
0807 08:45:42.754230 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:42.754289 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260807 08:45:42.855630 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 12769
I20260807 08:45:42.864392 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 12963
I20260807 08:45:42.872478 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 13118
2026-08-07T08:45:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18983 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:45:42 dist-test-slave-9zdj krb5kdc[13345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:45:42 dist-test-slave-9zdj krb5kdc[13345](info): set up 2 sockets
Aug 07 08:45:42 dist-test-slave-9zdj krb5kdc[13345](info): commencing operation
krb5kdc: starting...
W20260807 08:45:44.059923 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:45:44 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:45:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:45:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:45:44.218781 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:39241
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:39241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:44.312105 13359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:44.312366 13359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:44.312398 13359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:44.313742 13359 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 08:45:44.313802 13359 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 08:45:44.313814 13359 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 08:45:44.313833 13359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:45:44.313848 13359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:44.313861 13359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:44.313874 13359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:45:44.313885 13359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:45:44.315562 13359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:39241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:39241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:44.315909 13359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:44.316216 13359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:44.322180 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:44.322189 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:44.322193 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:44.322574 13359 server_base.cc:1034] running on GCE node
I20260807 08:45:44.322956 13359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:44.323342 13359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:44.324542 13359 hybrid_clock.cc:648] HybridClock initialized: now 1786092344324510 us; error 45 us; skew 500 ppm
Aug 07 08:45:44 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092344, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:44.335027 13359 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:45:44.335713 13359 webserver.cc:460] Webserver started at http://127.28.108.190:37239/ using document root <none> and password file <none>
I20260807 08:45:44.336038 13359 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:44.336093 13359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:44.336259 13359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:44.337478 13359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "b5589d728d224773a6a639f0f21bfef6"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "52d627c6b4b3aa41293bc18bbb74d229"
server_key_iv: "e86acc85e3d5e96f251f7a01658bf5ca"
server_key_version: "encryptionkey@0"
I20260807 08:45:44.338039 13359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "b5589d728d224773a6a639f0f21bfef6"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "52d627c6b4b3aa41293bc18bbb74d229"
server_key_iv: "e86acc85e3d5e96f251f7a01658bf5ca"
server_key_version: "encryptionkey@0"
I20260807 08:45:44.339677 13359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:44.340845 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:44.341171 13359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:44.341429 13359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "b5589d728d224773a6a639f0f21bfef6"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "52d627c6b4b3aa41293bc18bbb74d229"
server_key_iv: "e86acc85e3d5e96f251f7a01658bf5ca"
server_key_version: "encryptionkey@0"
I20260807 08:45:44.341533 13359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:44.358247 13359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:44.361413 13359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:44.361624 13359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:44.366833 13359 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:39241
I20260807 08:45:44.366914 13437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:39241 every 8 connection(s)
I20260807 08:45:44.367380 13359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:45:44.368494 13439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:44.371562 13439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Bootstrap starting.
I20260807 08:45:44.372326 13439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:44.372720 13439 log.cc:824] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:44.373816 13439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: No bootstrap required, opened a new log
I20260807 08:45:44.375779 13439 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:45:44.375976 13439 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:44.376003 13439 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5589d728d224773a6a639f0f21bfef6, State: Initialized, Role: FOLLOWER
I20260807 08:45:44.376195 13439 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:45:44.376292 13439 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:45:44.376322 13439 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:45:44.376412 13439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:44.376885 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 13359
I20260807 08:45:44.377046 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:45:44.377300 29106 external_mini_cluster.cc:1428] Setting key 78fc0dec9e99806b0311eba1915ef803
I20260807 08:45:44.377326 13439 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:45:44.377530 13439 leader_election.cc:302] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5589d728d224773a6a639f0f21bfef6; no voters:
I20260807 08:45:44.377830 13439 leader_election.cc:288] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:45:44.377915 13443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:45:44.378309 13439 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:45:44.378435 13443 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 1 LEADER]: Becoming Leader. State: Replica: b5589d728d224773a6a639f0f21bfef6, State: Running, Role: LEADER
I20260807 08:45:44.378655 13443 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:45:44.379347 13447 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5589d728d224773a6a639f0f21bfef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } } }
I20260807 08:45:44.379455 13447 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:44.379598 13445 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5589d728d224773a6a639f0f21bfef6. Latest consensus state: current_term: 1 leader_uuid: "b5589d728d224773a6a639f0f21bfef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } } }
I20260807 08:45:44.379725 13445 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: This master's current role is: LEADER
I20260807 08:45:44.379738 13456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:45:44.380527 13456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:45:44.384749 13456 catalog_manager.cc:1347] Generated new cluster ID: f99a77caf11d46d99dc12c061ef7d97c
I20260807 08:45:44.384853 13456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:45:44 dist-test-slave-9zdj krb5kdc[13345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:44.394160 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:44.378038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.28.108.190:39241)
0807 08:45:44.378320 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:44.378333 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:44.378390 (+ 57us) server_negotiation.cc:407] Connection header received
0807 08:45:44.378543 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:44.378553 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:44.378624 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:44.379411 (+ 787us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:44.380022 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:44.382536 (+ 2514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:44.383828 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:44.384580 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:44.384710 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:44.388365 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:44.388372 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:44.388373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:44.388391 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:44.391813 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:44.392424 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:44.392429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:44.392429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:44.392585 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:44.392853 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:44.392854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:44.392855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:44.393330 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:44.393526 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:44.393698 (+ 172us) server_negotiation.cc:299] Negotiation successful
0807 08:45:44.393892 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":125,"threads_started":1}
I20260807 08:45:44.416371 13456 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:45:44.417291 13456 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:45:44.422845 13456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Generated new TSK 0
I20260807 08:45:44.423132 13456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:45:44.463866 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39241
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:44.561709 13470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:44.561967 13470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:44.561996 13470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:44.563383 13470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:44.563462 13470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:44.563494 13470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:45:44.565714 13470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:44.566109 13470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:44.566457 13470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:44.573453 13475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:44.573453 13477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:44.573453 13479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:44.573899 13470 server_base.cc:1034] running on GCE node
I20260807 08:45:44.574178 13470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:44.574503 13470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:44.575886 13470 hybrid_clock.cc:648] HybridClock initialized: now 1786092344575813 us; error 51 us; skew 500 ppm
Aug 07 08:45:44 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092344, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:44.580044 13470 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:45:44.580684 13470 webserver.cc:460] Webserver started at http://127.28.108.129:33575/ using document root <none> and password file <none>
I20260807 08:45:44.580961 13470 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:44.581037 13470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:44.581199 13470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:44.582540 13470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "fda9e80b479a4129b8b11c335b241d5b"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "7eb2e0c010507a04a67c6ca70398ad25"
server_key_iv: "4df07000902f497787fcad469b6dc065"
server_key_version: "encryptionkey@0"
I20260807 08:45:44.582981 13470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "fda9e80b479a4129b8b11c335b241d5b"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "7eb2e0c010507a04a67c6ca70398ad25"
server_key_iv: "4df07000902f497787fcad469b6dc065"
server_key_version: "encryptionkey@0"
I20260807 08:45:44.584745 13470 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 08:45:44.586067 13487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:44.586462 13470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:44.586745 13470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "fda9e80b479a4129b8b11c335b241d5b"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "7eb2e0c010507a04a67c6ca70398ad25"
server_key_iv: "4df07000902f497787fcad469b6dc065"
server_key_version: "encryptionkey@0"
I20260807 08:45:44.586875 13470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:44.605425 13470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:44.608400 13470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:44.608647 13470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:44.609018 13470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:44.609437 13470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:44.609503 13470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:44.609543 13470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:44.609576 13470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:44.617974 13470 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:35045
I20260807 08:45:44.618057 13614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:35045 every 8 connection(s)
I20260807 08:45:44.618475 13470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:45:44.622035 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 13470
I20260807 08:45:44.622154 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:45:44.622465 29106 external_mini_cluster.cc:1428] Setting key 5498caea3a7a502e8c56468d29b2870f
Aug 07 08:45:44 dist-test-slave-9zdj krb5kdc[13345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092344, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:44.631992 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:44.619457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:54951 (local address 127.28.108.190:39241)
0807 08:45:44.619554 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:44.619556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:44.619728 (+ 172us) server_negotiation.cc:407] Connection header received
0807 08:45:44.620404 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:44.620405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:44.620423 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:44.620496 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:44.621822 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:44.622602 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:44.624184 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:44.624405 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:44.624473 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:44.627634 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:44.627636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:44.627637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:44.627648 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:44.629871 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:44.630601 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:44.630603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:44.630603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:44.630688 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:44.631075 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:44.631076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:44.631077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:44.631287 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:44.631376 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:44.631752 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 08:45:44.631865 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 08:45:44.632157 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:44.619363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:54951)
0807 08:45:44.619551 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:44.619600 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:44.620161 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:44.620638 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:44.620642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:44.621306 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:44.621539 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:44.621544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:44.622756 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:44.622758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:44.624058 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:44.624062 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:44.624519 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:44.624521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:44.624703 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:44.625508 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:44.625521 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:44.627479 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:44.630012 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:44.630025 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:44.630028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:44.630491 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:44.630812 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:44.630813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:44.630814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:44.630974 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:44.631392 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:44.631402 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:44.631563 (+ 161us) client_negotiation.cc:769] Sending connection context
0807 08:45:44.631680 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 08:45:44.631901 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260807 08:45:44.632874 13615 heartbeater.cc:347] Connected to a master server at 127.28.108.190:39241
I20260807 08:45:44.633060 13615 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:44.633441 13615 heartbeater.cc:511] Master 127.28.108.190:39241 requested a full tablet report, sending...
I20260807 08:45:44.634419 13395 ts_manager.cc:194] Registered new tserver with Master: fda9e80b479a4129b8b11c335b241d5b (127.28.108.129:35045)
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
I20260807 08:45:44.635279 13395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:54951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:45:44.665061 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39241
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:44.757694 13621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:44.758018 13621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:44.758067 13621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:44.759600 13621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:44.759864 13621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:44.759960 13621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:45:44.762291 13621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:44.762674 13621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:44.763000 13621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:44.769841 13628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:44.769785 13630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:44.769786 13627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:44.769896 13621 server_base.cc:1034] running on GCE node
I20260807 08:45:44.770453 13621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:44.770781 13621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:44.771983 13621 hybrid_clock.cc:648] HybridClock initialized: now 1786092344771956 us; error 39 us; skew 500 ppm
Aug 07 08:45:44 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092344, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:44.776058 13621 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:45:44.776885 13621 webserver.cc:460] Webserver started at http://127.28.108.130:34063/ using document root <none> and password file <none>
I20260807 08:45:44.777230 13621 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:44.777302 13621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:44.777446 13621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:44.778919 13621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "87759687187240caa1be334c4e737e48"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "7e98c4cc607e1ee9757c4d94cbc17149"
server_key_iv: "ea415a610174db03de57cbdbc83ad67b"
server_key_version: "encryptionkey@0"
I20260807 08:45:44.779434 13621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "87759687187240caa1be334c4e737e48"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "7e98c4cc607e1ee9757c4d94cbc17149"
server_key_iv: "ea415a610174db03de57cbdbc83ad67b"
server_key_version: "encryptionkey@0"
I20260807 08:45:44.781375 13621 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:45:44.782744 13638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:44.783000 13621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:44.783113 13621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "87759687187240caa1be334c4e737e48"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "7e98c4cc607e1ee9757c4d94cbc17149"
server_key_iv: "ea415a610174db03de57cbdbc83ad67b"
server_key_version: "encryptionkey@0"
I20260807 08:45:44.783193 13621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:44.823640 13621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:44.826805 13621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:44.827037 13621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:44.827332 13621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:44.827739 13621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:44.827813 13621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:44.827854 13621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:44.827886 13621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:44.834786 13621 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:36607
I20260807 08:45:44.834920 13771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:36607 every 8 connection(s)
I20260807 08:45:44.835269 13621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:45:44.843940 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 13621
I20260807 08:45:44.844094 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
Aug 07 08:45:44 dist-test-slave-9zdj krb5kdc[13345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092344, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:44.844352 29106 external_mini_cluster.cc:1428] Setting key 54b2eee64a5434c35f5667bee1eb5b63
I20260807 08:45:44.849148 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:44.836248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:46217 (local address 127.28.108.190:39241)
0807 08:45:44.836338 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:44.836341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:44.836663 (+ 322us) server_negotiation.cc:407] Connection header received
0807 08:45:44.837280 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:44.837282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:44.837296 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:44.837383 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:44.838671 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:44.839249 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:44.840781 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:44.841217 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:44.841297 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:44.844962 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:44.844964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:44.844964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:44.844974 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:44.847018 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:44.847758 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:44.847761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:44.847761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:44.847934 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:44.848263 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:44.848264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:44.848264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:44.848462 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:44.848555 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:44.848877 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 08:45:44.849005 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 08:45:44.849181 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:44.836132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:46217)
0807 08:45:44.836454 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:44.836519 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:44.837116 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:44.837493 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:44.837497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:44.838320 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:44.838547 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:44.838552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:44.839392 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:44.839393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:44.840643 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:44.840647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:44.841316 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:44.841318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:44.841493 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:44.842391 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:44.842403 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:44.844830 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:44.847142 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:44.847159 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:44.847163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:44.847610 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:44.848030 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:44.848031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:44.848032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:44.848182 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:44.848550 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:44.848557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:44.848715 (+ 158us) client_negotiation.cc:769] Sending connection context
0807 08:45:44.848820 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 08:45:44.848985 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":79,"threads_started":1}
I20260807 08:45:44.849839 13772 heartbeater.cc:347] Connected to a master server at 127.28.108.190:39241
I20260807 08:45:44.849996 13772 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:44.850244 13772 heartbeater.cc:511] Master 127.28.108.190:39241 requested a full tablet report, sending...
I20260807 08:45:44.850782 13395 ts_manager.cc:194] Registered new tserver with Master: 87759687187240caa1be334c4e737e48 (127.28.108.130:36607)
I20260807 08:45:44.851262 13395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:46217
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:45:44.882896 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39241
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:45:44.981920 13777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:45:44.982162 13777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:45:44.982191 13777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:45:44.983413 13777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:45:44.983457 13777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:45:44.983484 13777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:45:44.985070 13777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:45:44.985368 13777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:45:44.985575 13777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:45:44.992064 13784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:45:44.992003 13783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:44.992004 13777 server_base.cc:1034] running on GCE node
W20260807 08:45:44.992111 13786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:45:44.992545 13777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:45:44.992841 13777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:45:44.994002 13777 hybrid_clock.cc:648] HybridClock initialized: now 1786092344993974 us; error 46 us; skew 500 ppm
Aug 07 08:45:44 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092344, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:45:44.997035 13777 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:45:44.997658 13777 webserver.cc:460] Webserver started at http://127.28.108.131:44047/ using document root <none> and password file <none>
I20260807 08:45:44.997949 13777 fs_manager.cc:359] Metadata directory not provided
I20260807 08:45:44.997999 13777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:45:44.998188 13777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:45:44.999369 13777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "3c66605377eb4e0f97023d5f124a882e"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "488ca65a7729a4a614a7758637ab3011"
server_key_iv: "3af940c186aa838908aacc0c110a6904"
server_key_version: "encryptionkey@0"
I20260807 08:45:44.999766 13777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "3c66605377eb4e0f97023d5f124a882e"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "488ca65a7729a4a614a7758637ab3011"
server_key_iv: "3af940c186aa838908aacc0c110a6904"
server_key_version: "encryptionkey@0"
I20260807 08:45:45.001508 13777 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 08:45:45.002972 13794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:45.003392 13777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:45:45.003572 13777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "3c66605377eb4e0f97023d5f124a882e"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "488ca65a7729a4a614a7758637ab3011"
server_key_iv: "3af940c186aa838908aacc0c110a6904"
server_key_version: "encryptionkey@0"
I20260807 08:45:45.003666 13777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:45:45.054920 13777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:45:45.058744 13777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:45:45.058995 13777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:45:45.059320 13777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:45:45.059731 13777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:45:45.059800 13777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:45.059854 13777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:45:45.059891 13777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:45:45.069262 13777 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:35531
I20260807 08:45:45.069312 13920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:35531 every 8 connection(s)
I20260807 08:45:45.069969 13777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:45:45.070226 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 13777
I20260807 08:45:45.070349 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:45:45.070578 29106 external_mini_cluster.cc:1428] Setting key 62a68c705d038e8c3e8d5fac1d811a3b
Aug 07 08:45:45 dist-test-slave-9zdj krb5kdc[13345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092344, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:45:45.082743 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.070951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:47399 (local address 127.28.108.190:39241)
0807 08:45:45.071005 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:45.071008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:45.071502 (+ 494us) server_negotiation.cc:407] Connection header received
0807 08:45:45.072216 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:45.072217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:45.072232 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:45.072278 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:45.073452 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.074112 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.075319 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.075572 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.075675 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.078538 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:45.078543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:45.078543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:45.078558 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:45.080670 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:45.081184 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:45.081186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:45.081186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:45.081258 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:45.081573 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:45.081574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:45.081574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:45.081779 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:45.081872 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:45.082435 (+ 563us) server_negotiation.cc:299] Negotiation successful
0807 08:45:45.082599 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 08:45:45.082746 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.070930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:47399)
0807 08:45:45.071363 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:45.071422 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:45.072077 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:45.072394 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:45.072399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:45.073079 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:45:45.073331 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.073337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.074240 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.074241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.075202 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.075205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.075681 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.075682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.075861 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.076571 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:45:45.076580 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:45:45.078410 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:45:45.080785 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:45.080805 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:45.080808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:45.081088 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:45.081343 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:45:45.081344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:45:45.081345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:45:45.081496 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:45:45.081877 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:45:45.081888 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:45:45.082060 (+ 172us) client_negotiation.cc:769] Sending connection context
0807 08:45:45.082204 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 08:45:45.082439 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":143,"threads_started":1}
I20260807 08:45:45.083922 13921 heartbeater.cc:347] Connected to a master server at 127.28.108.190:39241
I20260807 08:45:45.084048 13921 heartbeater.cc:464] Registering TS with master...
I20260807 08:45:45.084249 13921 heartbeater.cc:511] Master 127.28.108.190:39241 requested a full tablet report, sending...
I20260807 08:45:45.084798 13395 ts_manager.cc:194] Registered new tserver with Master: 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531)
I20260807 08:45:45.085384 13395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:47399
I20260807 08:45:45.094708 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:45:45.104722 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.096524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.28.108.190:39241)
0807 08:45:45.096604 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:45.096606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:45.096714 (+ 108us) server_negotiation.cc:407] Connection header received
0807 08:45:45.096765 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:45.096766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:45.096777 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:45.096834 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:45:45.097507 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.098142 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.099055 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.099258 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.099332 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.100986 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:45:45.100988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:45:45.100988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:45:45.100998 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:45:45.102962 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:45.103534 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:45.103536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:45.103536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:45.103611 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:45:45.103933 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:45:45.103934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:45:45.103935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:45:45.104135 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:45:45.104238 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:45.104506 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 08:45:45.104603 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 08:45:45.107523 13395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:45:45.109664 13395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:45:45.118211 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.113416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:35531 (local address 127.0.0.1:59216)
0807 08:45:45.114106 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:45.114118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:45.114171 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:45.114560 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:45.114561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:45.114564 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:45.114615 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.114616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.116107 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.116108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.116836 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.116838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.117916 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.117917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.117968 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.117971 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:45:45.118036 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 08:45:45.118099 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":104,"threads_started":1}
I20260807 08:45:45.118541 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.113148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:36607 (local address 127.0.0.1:45402)
0807 08:45:45.113369 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:45.113382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:45.113470 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:45.114053 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:45.114057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:45.114061 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:45.114203 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.114206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.115619 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.115620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.116752 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.116756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.117810 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.117811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.117943 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.117962 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 08:45:45.118036 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 08:45:45.118337 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":38,"thread_start_us":116,"threads_started":1}
I20260807 08:45:45.118573 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.113819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.28.108.131:35531)
0807 08:45:45.114052 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:45.114057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:45.114246 (+ 189us) server_negotiation.cc:407] Connection header received
0807 08:45:45.114321 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:45.114324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:45.114342 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:45.114530 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:45.114815 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.115999 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.116965 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.117718 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.117817 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.117882 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:45.118428 (+ 546us) server_negotiation.cc:299] Negotiation successful
0807 08:45:45.118485 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
I20260807 08:45:45.118902 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.113362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.28.108.130:36607)
0807 08:45:45.113806 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:45.113813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:45.113825 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:45:45.113879 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:45.113882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:45.113903 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:45.114488 (+ 585us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:45.114619 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.115491 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.116867 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.117713 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.117798 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.117860 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:45.118754 (+ 894us) server_negotiation.cc:299] Negotiation successful
0807 08:45:45.118812 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":94,"threads_started":1}
I20260807 08:45:45.119372 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.113142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:35045 (local address 127.0.0.1:39076)
0807 08:45:45.113303 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:45.113363 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:45.113438 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:45.114107 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:45.114108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:45.114110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:45.114203 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.114208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.115622 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.115623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.117927 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.117930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.118975 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.118976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.119072 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.119077 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:45:45.119223 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 08:45:45.119278 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":83,"thread_start_us":99,"threads_started":1}
I20260807 08:45:45.119357 13848 tablet_service.cc:1467] Processing CreateTablet for tablet 6bcb5c652ac64953ab0d59408bb1a116 (DEFAULT_TABLE table=test-table [id=c9be7ee8b788478d90b50df51f1a1739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:45.119501 13696 tablet_service.cc:1467] Processing CreateTablet for tablet 6bcb5c652ac64953ab0d59408bb1a116 (DEFAULT_TABLE table=test-table [id=c9be7ee8b788478d90b50df51f1a1739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:45.119711 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.113684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.28.108.129:35045)
0807 08:45:45.113893 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:45.113898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:45.113910 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:45:45.113958 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:45.113961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:45.113977 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:45.114063 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:45.114302 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.115491 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.118027 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.118784 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.118866 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.118929 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:45.119553 (+ 624us) server_negotiation.cc:299] Negotiation successful
0807 08:45:45.119621 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":60,"threads_started":1}
I20260807 08:45:45.119797 13696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bcb5c652ac64953ab0d59408bb1a116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:45.119789 13848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bcb5c652ac64953ab0d59408bb1a116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:45.120416 13541 tablet_service.cc:1467] Processing CreateTablet for tablet 6bcb5c652ac64953ab0d59408bb1a116 (DEFAULT_TABLE table=test-table [id=c9be7ee8b788478d90b50df51f1a1739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:45:45.120779 13541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bcb5c652ac64953ab0d59408bb1a116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:45:45.124361 13941 tablet_bootstrap.cc:492] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48: Bootstrap starting.
I20260807 08:45:45.125029 13942 tablet_bootstrap.cc:492] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Bootstrap starting.
I20260807 08:45:45.125700 13940 tablet_bootstrap.cc:492] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e: Bootstrap starting.
I20260807 08:45:45.125903 13942 tablet_bootstrap.cc:654] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:45.125905 13941 tablet_bootstrap.cc:654] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:45.126569 13940 tablet_bootstrap.cc:654] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e: Neither blocks nor log segments found. Creating new log.
I20260807 08:45:45.126645 13942 log.cc:824] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:45.126642 13941 log.cc:824] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:45.127357 13940 log.cc:824] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e: Log is configured to *not* fsync() on all Append() calls
I20260807 08:45:45.127803 13942 tablet_bootstrap.cc:492] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: No bootstrap required, opened a new log
I20260807 08:45:45.127928 13942 ts_tablet_manager.cc:1397] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:45:45.127969 13941 tablet_bootstrap.cc:492] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48: No bootstrap required, opened a new log
I20260807 08:45:45.128062 13941 ts_tablet_manager.cc:1397] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260807 08:45:45.128824 13940 tablet_bootstrap.cc:492] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e: No bootstrap required, opened a new log
I20260807 08:45:45.128937 13940 ts_tablet_manager.cc:1397] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:45:45.131004 13941 raft_consensus.cc:348] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } }
I20260807 08:45:45.131263 13941 raft_consensus.cc:374] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:45.131309 13941 raft_consensus.cc:729] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87759687187240caa1be334c4e737e48, State: Initialized, Role: FOLLOWER
I20260807 08:45:45.131454 13941 consensus_queue.cc:260] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } }
I20260807 08:45:45.131513 13942 raft_consensus.cc:348] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } }
I20260807 08:45:45.131702 13942 raft_consensus.cc:374] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:45.131739 13942 raft_consensus.cc:729] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda9e80b479a4129b8b11c335b241d5b, State: Initialized, Role: FOLLOWER
I20260807 08:45:45.131814 13941 ts_tablet_manager.cc:1428] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260807 08:45:45.131901 13942 consensus_queue.cc:260] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } }
I20260807 08:45:45.131953 13772 heartbeater.cc:503] Master 127.28.108.190:39241 was elected leader, sending a full tablet report...
I20260807 08:45:45.132091 13940 raft_consensus.cc:348] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } }
I20260807 08:45:45.132342 13942 ts_tablet_manager.cc:1428] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260807 08:45:45.132480 13615 heartbeater.cc:503] Master 127.28.108.190:39241 was elected leader, sending a full tablet report...
I20260807 08:45:45.132305 13940 raft_consensus.cc:374] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:45:45.134042 13940 raft_consensus.cc:729] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c66605377eb4e0f97023d5f124a882e, State: Initialized, Role: FOLLOWER
I20260807 08:45:45.134208 13940 consensus_queue.cc:260] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } }
I20260807 08:45:45.134609 13940 ts_tablet_manager.cc:1428] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260807 08:45:45.134609 13921 heartbeater.cc:503] Master 127.28.108.190:39241 was elected leader, sending a full tablet report...
I20260807 08:45:45.225749 13947 raft_consensus.cc:482] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:45:45.225916 13947 raft_consensus.cc:504] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } }
I20260807 08:45:45.226486 13947 leader_election.cc:288] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531), 87759687187240caa1be334c4e737e48 (127.28.108.130:36607)
I20260807 08:45:45.230123 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.226870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:56289 (local address 127.28.108.130:36607)
0807 08:45:45.226927 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:45.226930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:45.227059 (+ 129us) server_negotiation.cc:407] Connection header received
0807 08:45:45.227137 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:45.227138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:45.227149 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:45.227248 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:45.227431 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.227978 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.229155 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.229604 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.229755 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.229767 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:45.229979 (+ 212us) server_negotiation.cc:299] Negotiation successful
0807 08:45:45.230030 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 08:45:45.230127 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.226914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:54619 (local address 127.28.108.131:35531)
0807 08:45:45.226968 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:45.226970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:45.227063 (+ 93us) server_negotiation.cc:407] Connection header received
0807 08:45:45.227122 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:45.227123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:45.227134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:45.227224 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:45:45.227511 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.228043 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.229162 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.229604 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.229752 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.229775 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:45.229978 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 08:45:45.230029 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260807 08:45:45.230194 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.226747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:35531 (local address 127.28.108.129:54619)
0807 08:45:45.226950 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:45.226964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:45.227053 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:45.227231 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:45.227232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:45.227235 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:45.227411 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.227414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.228146 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.228147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.229042 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.229045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.229754 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.229755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.229870 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.229880 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 08:45:45.229970 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 08:45:45.230027 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
I20260807 08:45:45.230197 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.226750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:36607 (local address 127.28.108.129:56289)
0807 08:45:45.226954 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:45.226966 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:45:45.227070 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:45:45.227258 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:45:45.227259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:45:45.227261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:45:45.227343 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.227346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.228086 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.228087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.229030 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:45:45.229033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.229762 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.229764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:45:45.229879 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.229887 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 08:45:45.229973 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 08:45:45.230031 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":113,"threads_started":1}
I20260807 08:45:45.230681 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcb5c652ac64953ab0d59408bb1a116" candidate_uuid: "fda9e80b479a4129b8b11c335b241d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c66605377eb4e0f97023d5f124a882e" is_pre_election: true
I20260807 08:45:45.230896 13871 raft_consensus.cc:2454] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fda9e80b479a4129b8b11c335b241d5b in term 0.
I20260807 08:45:45.230818 13717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcb5c652ac64953ab0d59408bb1a116" candidate_uuid: "fda9e80b479a4129b8b11c335b241d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87759687187240caa1be334c4e737e48" is_pre_election: true
I20260807 08:45:45.231029 13717 raft_consensus.cc:2454] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fda9e80b479a4129b8b11c335b241d5b in term 0.
I20260807 08:45:45.231268 13491 leader_election.cc:302] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c66605377eb4e0f97023d5f124a882e, fda9e80b479a4129b8b11c335b241d5b; no voters:
I20260807 08:45:45.231480 13947 raft_consensus.cc:2781] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:45:45.231567 13947 raft_consensus.cc:482] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:45:45.231590 13947 raft_consensus.cc:3037] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:45.232657 13947 raft_consensus.cc:504] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } }
I20260807 08:45:45.232856 13947 leader_election.cc:288] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 1 election: Requested vote from peers 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531), 87759687187240caa1be334c4e737e48 (127.28.108.130:36607)
I20260807 08:45:45.233071 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcb5c652ac64953ab0d59408bb1a116" candidate_uuid: "fda9e80b479a4129b8b11c335b241d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c66605377eb4e0f97023d5f124a882e"
I20260807 08:45:45.233206 13871 raft_consensus.cc:3037] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:45.233338 13717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bcb5c652ac64953ab0d59408bb1a116" candidate_uuid: "fda9e80b479a4129b8b11c335b241d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87759687187240caa1be334c4e737e48"
I20260807 08:45:45.233464 13717 raft_consensus.cc:3037] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:45:45.234263 13871 raft_consensus.cc:2454] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda9e80b479a4129b8b11c335b241d5b in term 1.
I20260807 08:45:45.234427 13717 raft_consensus.cc:2454] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda9e80b479a4129b8b11c335b241d5b in term 1.
I20260807 08:45:45.234524 13491 leader_election.cc:302] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c66605377eb4e0f97023d5f124a882e, fda9e80b479a4129b8b11c335b241d5b; no voters:
I20260807 08:45:45.234668 13947 raft_consensus.cc:2781] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:45:45.234905 13947 raft_consensus.cc:686] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 1 LEADER]: Becoming Leader. State: Replica: fda9e80b479a4129b8b11c335b241d5b, State: Running, Role: LEADER
I20260807 08:45:45.235046 13947 consensus_queue.cc:237] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } }
I20260807 08:45:45.236140 13394 catalog_manager.cc:5495] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b reported cstate change: term changed from 0 to 1, leader changed from <none> to fda9e80b479a4129b8b11c335b241d5b (127.28.108.129). New cstate: current_term: 1 leader_uuid: "fda9e80b479a4129b8b11c335b241d5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } health_report { overall_health: UNKNOWN } } }
I20260807 08:45:45.268291 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.265189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.28.108.129:35045)
0807 08:45:45.265366 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 08:45:45.265369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:45:45.265378 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:45:45.265423 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:45:45.265424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:45:45.265445 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:45:45.265520 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:45:45.265742 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.266245 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.267156 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:45:45.267396 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:45:45.267473 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:45:45.267723 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:45:45.267850 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:45:45.267939 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 08:45:45.267999 (+ 60us) server_negotiation.cc:299] Negotiation successful
0807 08:45:45.268037 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 08:45:45.270700 13871 raft_consensus.cc:1261] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e [term 1 FOLLOWER]: Refusing update from remote peer fda9e80b479a4129b8b11c335b241d5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:45:45.270845 13717 raft_consensus.cc:1261] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48 [term 1 FOLLOWER]: Refusing update from remote peer fda9e80b479a4129b8b11c335b241d5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:45:45.271147 13947 consensus_queue.cc:1035] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:45:45.271437 13954 consensus_queue.cc:1035] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:45:45.274904 13957 mvcc.cc:205] Tried to move back new op lower bound from 7315834246225403904 to 7315834246083809280. Current Snapshot: MvccSnapshot[applied={T|T < 7315834246225403904}]
I20260807 08:45:45.275411 13958 mvcc.cc:205] Tried to move back new op lower bound from 7315834246225403904 to 7315834246083809280. Current Snapshot: MvccSnapshot[applied={T|T < 7315834246225403904}]
I20260807 08:45:45.276049 13960 mvcc.cc:205] Tried to move back new op lower bound from 7315834246225403904 to 7315834246083809280. Current Snapshot: MvccSnapshot[applied={T|T < 7315834246225403904}]
I20260807 08:45:45.278368 13393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:45:45.278506 13393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:45:45.279490 13393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Sending DeleteTablet for 3 replicas of tablet 6bcb5c652ac64953ab0d59408bb1a116
I20260807 08:45:45.279950 13696 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bcb5c652ac64953ab0d59408bb1a116 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:45 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:45402
I20260807 08:45:45.279934 13541 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bcb5c652ac64953ab0d59408bb1a116 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:45 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:39076
I20260807 08:45:45.280270 13967 tablet_replica.cc:330] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: stopping tablet replica
I20260807 08:45:45.280285 13848 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bcb5c652ac64953ab0d59408bb1a116 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:45 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:59216
I20260807 08:45:45.280383 13967 raft_consensus.cc:2229] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:45:45.280539 13967 raft_consensus.cc:2258] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:45:45.280550 13969 tablet_replica.cc:330] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e: stopping tablet replica
I20260807 08:45:45.280668 13969 raft_consensus.cc:2229] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:45:45.280719 13968 tablet_replica.cc:330] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48: stopping tablet replica
I20260807 08:45:45.280795 13969 raft_consensus.cc:2258] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:45:45.280844 13968 raft_consensus.cc:2229] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:45:45.280846 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 13470
I20260807 08:45:45.280958 13968 raft_consensus.cc:2258] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:45:45.284070 13968 ts_tablet_manager.cc:1905] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:45:45.286733 13968 ts_tablet_manager.cc:1918] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:45:45.286886 13968 log.cc:1196] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/6bcb5c652ac64953ab0d59408bb1a116
I20260807 08:45:45.287201 13968 ts_tablet_manager.cc:1939] T 6bcb5c652ac64953ab0d59408bb1a116 P 87759687187240caa1be334c4e737e48: Deleting consensus metadata
I20260807 08:45:45.287561 13969 ts_tablet_manager.cc:1905] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:45:45.287855 13381 catalog_manager.cc:4841] TS 87759687187240caa1be334c4e737e48 (127.28.108.130:36607): tablet 6bcb5c652ac64953ab0d59408bb1a116 (table test-table [id=c9be7ee8b788478d90b50df51f1a1739]) successfully deleted
I20260807 08:45:45.289850 13969 ts_tablet_manager.cc:1918] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:45:45.289960 13969 log.cc:1196] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/6bcb5c652ac64953ab0d59408bb1a116
I20260807 08:45:45.290278 13969 ts_tablet_manager.cc:1939] T 6bcb5c652ac64953ab0d59408bb1a116 P 3c66605377eb4e0f97023d5f124a882e: Deleting consensus metadata
I20260807 08:45:45.290941 13382 catalog_manager.cc:4841] TS 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531): tablet 6bcb5c652ac64953ab0d59408bb1a116 (table test-table [id=c9be7ee8b788478d90b50df51f1a1739]) successfully deleted
W20260807 08:45:45.291640 13382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:35045 (error 108)
I20260807 08:45:45.291903 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 13621
I20260807 08:45:45.294351 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:45:45.294175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:35045 (local address 127.0.0.1:39104)
0807 08:45:45.294247 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0807 08:45:45.294300 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:35045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 08:45:45.294505 13382 catalog_manager.cc:4584] TS fda9e80b479a4129b8b11c335b241d5b (127.28.108.129:35045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6bcb5c652ac64953ab0d59408bb1a116: Network error: Client connection negotiation failed: client connection to 127.28.108.129:35045: connect: Connection refused (error 111)
I20260807 08:45:45.299880 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 13777
I20260807 08:45:45.308363 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 13359
I20260807 08:46:05.318408 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:39241
--webserver_interface=127.28.108.190
--webserver_port=37239
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:39241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:05.410835 13971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:05.411154 13971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:05.411198 13971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:05.412583 13971 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 08:46:05.412690 13971 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 08:46:05.412719 13971 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 08:46:05.412741 13971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:05.412766 13971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:05.412830 13971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:05.412865 13971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:05.412896 13971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:05.414798 13971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:39241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:39241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:05.415169 13971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:05.415453 13971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:05.428892 13977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:05.429033 13978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:05.428843 13980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:05.429129 13971 server_base.cc:1034] running on GCE node
I20260807 08:46:05.429970 13971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:05.430420 13971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:05.431744 13971 hybrid_clock.cc:648] HybridClock initialized: now 1786092365431659 us; error 71 us; skew 500 ppm
Aug 07 08:46:05 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092365, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:05.442725 13971 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:46:05.443447 13971 webserver.cc:460] Webserver started at http://127.28.108.190:37239/ using document root <none> and password file <none>
I20260807 08:46:05.443802 13971 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:05.443879 13971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:05.446115 13971 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260807 08:46:05.447520 13987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:05.447875 13971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:05.447978 13971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "b5589d728d224773a6a639f0f21bfef6"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "52d627c6b4b3aa41293bc18bbb74d229"
server_key_iv: "e86acc85e3d5e96f251f7a01658bf5ca"
server_key_version: "encryptionkey@0"
I20260807 08:46:05.448455 13971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:05.462055 13971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:05.466277 13971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:05.466447 13971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:05.471732 13971 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:39241
I20260807 08:46:05.471958 14048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:39241 every 8 connection(s)
I20260807 08:46:05.472350 13971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:05.474572 14049 sys_catalog.cc:263] Verifying existing consensus state
I20260807 08:46:05.476030 14049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Bootstrap starting.
I20260807 08:46:05.477042 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 13971
I20260807 08:46:05.477491 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:35045
--local_ip_for_outbound_sockets=127.28.108.129
--tserver_master_addrs=127.28.108.190:39241
--webserver_port=33575
--webserver_interface=127.28.108.129
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:46:05.490609 14049 log.cc:824] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:05.494640 14049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:46:05.495121 14049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Bootstrap complete.
I20260807 08:46:05.498850 14049 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:46:05.499245 14049 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5589d728d224773a6a639f0f21bfef6, State: Initialized, Role: FOLLOWER
I20260807 08:46:05.499470 14049 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:46:05.499552 14049 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:05.499603 14049 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:05.499644 14049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 1 FOLLOWER]: Advancing to term 2
I20260807 08:46:05.501209 14049 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:46:05.501360 14049 leader_election.cc:302] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5589d728d224773a6a639f0f21bfef6; no voters:
I20260807 08:46:05.501570 14049 leader_election.cc:288] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 08:46:05.501725 14054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 08:46:05.501889 14049 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:05.501926 14054 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 2 LEADER]: Becoming Leader. State: Replica: b5589d728d224773a6a639f0f21bfef6, State: Running, Role: LEADER
I20260807 08:46:05.502063 14054 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:46:05.502586 14055 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b5589d728d224773a6a639f0f21bfef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } } }
I20260807 08:46:05.502672 14055 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:05.503011 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5589d728d224773a6a639f0f21bfef6. Latest consensus state: current_term: 2 leader_uuid: "b5589d728d224773a6a639f0f21bfef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } } }
I20260807 08:46:05.503186 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:05.503068 14066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:05.504392 14066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:05.504503 14066 catalog_manager.cc:1259] Loaded cluster ID: f99a77caf11d46d99dc12c061ef7d97c
I20260807 08:46:05.504534 14066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:46:05.505887 14066 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:05.506136 14066 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Loaded TSK: 0
I20260807 08:46:05.515144 14066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Generated new TSK 1
I20260807 08:46:05.515305 14066 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 08:46:05.592092 14051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:05.592420 14051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:05.592473 14051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:05.594030 14051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:05.594143 14051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:05.594198 14051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:46:05.596787 14051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:35045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=33575
--tserver_master_addrs=127.28.108.190:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:05.597177 14051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:05.597522 14051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:05.604116 14087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:05.604031 14082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:05.604031 14084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:05.604676 14051 server_base.cc:1034] running on GCE node
I20260807 08:46:05.604918 14051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:05.605283 14051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:05.606489 14051 hybrid_clock.cc:648] HybridClock initialized: now 1786092365606464 us; error 37 us; skew 500 ppm
Aug 07 08:46:05 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092365, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:05.610023 14051 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:46:05.610965 14051 webserver.cc:460] Webserver started at http://127.28.108.129:33575/ using document root <none> and password file <none>
I20260807 08:46:05.611295 14051 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:05.611403 14051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:05.613123 14051 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:46:05.614153 14095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:05.614408 14051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:05.614533 14051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "fda9e80b479a4129b8b11c335b241d5b"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "7eb2e0c010507a04a67c6ca70398ad25"
server_key_iv: "4df07000902f497787fcad469b6dc065"
server_key_version: "encryptionkey@0"
I20260807 08:46:05.615031 14051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:05.631668 14051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:05.635078 14051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:05.635308 14051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:05.635664 14051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:05.636373 14106 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 08:46:05.638200 14051 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 08:46:05.638273 14051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260807 08:46:05.638312 14051 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 08:46:05.639405 14051 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 08:46:05.639464 14051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260807 08:46:05.639508 14106 tablet_bootstrap.cc:492] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Bootstrap starting.
I20260807 08:46:05.649751 14051 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:35045
I20260807 08:46:05.650197 14236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:35045 every 8 connection(s)
I20260807 08:46:05.650437 14051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:46:05.654702 14106 log.cc:824] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:05.657943 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 14051
I20260807 08:46:05.658072 14106 tablet_bootstrap.cc:492] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:46:05.658334 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:36607
--local_ip_for_outbound_sockets=127.28.108.130
--tserver_master_addrs=127.28.108.190:39241
--webserver_port=34063
--webserver_interface=127.28.108.130
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:46:05.658682 14106 tablet_bootstrap.cc:492] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Bootstrap complete.
I20260807 08:46:05.660187 14106 ts_tablet_manager.cc:1397] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.010s
Aug 07 08:46:05 dist-test-slave-9zdj krb5kdc[13345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092365, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:05.662726 14106 raft_consensus.cc:348] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } }
I20260807 08:46:05.663223 14106 raft_consensus.cc:729] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda9e80b479a4129b8b11c335b241d5b, State: Initialized, Role: FOLLOWER
I20260807 08:46:05.663434 14106 consensus_queue.cc:260] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } } peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } }
I20260807 08:46:05.663820 14106 ts_tablet_manager.cc:1428] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20260807 08:46:05.670112 14239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:05.651389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:47737)
0807 08:46:05.651614 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:05.651680 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:05.652579 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:05.653253 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:05.653259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:05.653918 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:05.654309 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:05.654321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.656874 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.656876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:05.658422 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:05.658428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.659975 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.659976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:05.660197 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:05.660962 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:05.660989 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:05.663561 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:05.667097 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:05.667112 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:05.667115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:05.667587 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:05.668056 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:05.668057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:05.668058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:05.668277 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:05.669113 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:05.669128 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:05.669400 (+ 272us) client_negotiation.cc:769] Sending connection context
0807 08:46:05.669535 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 08:46:05.669832 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":113,"threads_started":1}
I20260807 08:46:05.670244 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:05.651559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:47737 (local address 127.28.108.190:39241)
0807 08:46:05.651788 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:05.651799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:05.651859 (+ 60us) server_negotiation.cc:407] Connection header received
0807 08:46:05.652926 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:05.652935 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:05.653004 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:05.653167 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:05.654514 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.656670 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.658941 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.659813 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.660675 (+ 862us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:05.663895 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:05.663903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:05.663903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:05.663919 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:05.666939 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:05.667760 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:05.667765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:05.667766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:05.667953 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:05.668513 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:05.668515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:05.668515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:05.668992 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:05.669111 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:05.669742 (+ 631us) server_negotiation.cc:299] Negotiation successful
0807 08:46:05.669956 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":101,"threads_started":1}
I20260807 08:46:05.671940 14237 heartbeater.cc:347] Connected to a master server at 127.28.108.190:39241
I20260807 08:46:05.672080 14237 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:05.672310 14237 heartbeater.cc:511] Master 127.28.108.190:39241 requested a full tablet report, sending...
I20260807 08:46:05.673333 14008 ts_manager.cc:194] Registered new tserver with Master: fda9e80b479a4129b8b11c335b241d5b (127.28.108.129:35045)
I20260807 08:46:05.675068 14008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:47737
I20260807 08:46:05.676424 14237 heartbeater.cc:503] Master 127.28.108.190:39241 was elected leader, sending a full tablet report...
I20260807 08:46:05.682494 14247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:05.674814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34784 (local address 127.28.108.129:35045)
0807 08:46:05.675134 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:05.675141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:05.677925 (+ 2784us) server_negotiation.cc:407] Connection header received
0807 08:46:05.678034 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:05.678037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:05.678057 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:05.678161 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:05.678355 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.679429 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.681181 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.681993 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.682075 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:05.682122 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:05.682370 (+ 248us) server_negotiation.cc:299] Negotiation successful
0807 08:46:05.682423 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":138,"threads_started":1}
I20260807 08:46:05.682508 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:05.674273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:35045 (local address 127.0.0.1:34784)
0807 08:46:05.677764 (+ 3491us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:05.677823 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:05.677920 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:05.678179 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:05.678182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:05.678185 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:05.678271 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:05.678273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.679562 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.679563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:05.681037 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:05.681043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.682093 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.682094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:05.682204 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:05.682223 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 08:46:05.682313 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 08:46:05.682382 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3451,"thread_start_us":151,"threads_started":1}
I20260807 08:46:05.683130 14156 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bcb5c652ac64953ab0d59408bb1a116 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:45:45 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:34784
I20260807 08:46:05.683900 14248 tablet_replica.cc:330] stopping tablet replica
I20260807 08:46:05.684072 14248 raft_consensus.cc:2229] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:05.684295 14248 raft_consensus.cc:2258] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:05.684716 14248 ts_tablet_manager.cc:1905] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:05.687500 14248 ts_tablet_manager.cc:1918] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:05.687597 14248 log.cc:1196] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal/wals/6bcb5c652ac64953ab0d59408bb1a116
I20260807 08:46:05.688059 14248 ts_tablet_manager.cc:1939] T 6bcb5c652ac64953ab0d59408bb1a116 P fda9e80b479a4129b8b11c335b241d5b: Deleting consensus metadata
I20260807 08:46:05.688797 13993 catalog_manager.cc:4841] TS fda9e80b479a4129b8b11c335b241d5b (127.28.108.129:35045): tablet 6bcb5c652ac64953ab0d59408bb1a116 (table test-table [id=c9be7ee8b788478d90b50df51f1a1739]) successfully deleted
W20260807 08:46:05.759634 14244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:05.759939 14244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:05.759994 14244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:05.761551 14244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:05.761647 14244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:05.761686 14244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:46:05.763569 14244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:36607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=34063
--tserver_master_addrs=127.28.108.190:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:05.763872 14244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:05.764101 14244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:05.770743 14254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:05.770735 14256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:05.770735 14253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:05.770790 14244 server_base.cc:1034] running on GCE node
I20260807 08:46:05.771328 14244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:05.771653 14244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:05.772904 14244 hybrid_clock.cc:648] HybridClock initialized: now 1786092365772874 us; error 53 us; skew 500 ppm
Aug 07 08:46:05 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092365, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:05.776273 14244 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:46:05.776950 14244 webserver.cc:460] Webserver started at http://127.28.108.130:34063/ using document root <none> and password file <none>
I20260807 08:46:05.777267 14244 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:05.777359 14244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:05.779270 14244 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:05.780172 14265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:05.780491 14244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:05.780748 14244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "87759687187240caa1be334c4e737e48"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "7e98c4cc607e1ee9757c4d94cbc17149"
server_key_iv: "ea415a610174db03de57cbdbc83ad67b"
server_key_version: "encryptionkey@0"
I20260807 08:46:05.781191 14244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:05.812202 14244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:05.815584 14244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:05.815881 14244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:05.816195 14244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:05.816572 14244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:05.816632 14244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:05.816679 14244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:05.816708 14244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:05.823611 14244 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:36607
I20260807 08:46:05.823658 14387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:36607 every 8 connection(s)
I20260807 08:46:05.824100 14244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:46:05.830538 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 14244
I20260807 08:46:05.830912 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:35531
--local_ip_for_outbound_sockets=127.28.108.131
--tserver_master_addrs=127.28.108.190:39241
--webserver_port=44047
--webserver_interface=127.28.108.131
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 08:46:05 dist-test-slave-9zdj krb5kdc[13345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092365, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:05.837356 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:05.825037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:51287 (local address 127.28.108.190:39241)
0807 08:46:05.825164 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:05.825306 (+ 142us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:05.825486 (+ 180us) server_negotiation.cc:407] Connection header received
0807 08:46:05.826218 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:05.826220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:05.826239 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:05.826296 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:05.827605 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.828195 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.829430 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.829774 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.829865 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:05.833000 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:05.833005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:05.833005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:05.833019 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:05.834918 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:05.836003 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:05.836005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:05.836005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:05.836089 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:05.836386 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:05.836387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:05.836388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:05.836567 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:05.836657 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:05.837081 (+ 424us) server_negotiation.cc:299] Negotiation successful
0807 08:46:05.837224 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260807 08:46:05.837544 14392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:05.825023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:51287)
0807 08:46:05.825307 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:05.825369 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:05.826067 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:05.826381 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:05.826386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:05.827217 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:05.827461 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:05.827467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.828326 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.828327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:05.829293 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:05.829296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.829884 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.829885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:05.830121 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:05.830850 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:05.830861 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:05.832867 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:05.835369 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:05.835379 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:05.835382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:05.835889 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:05.836171 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:05.836172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:05.836172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:05.836310 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:05.836665 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:05.836672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:05.836902 (+ 230us) client_negotiation.cc:769] Sending connection context
0807 08:46:05.837019 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 08:46:05.837187 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":127,"threads_started":1}
I20260807 08:46:05.838282 14388 heartbeater.cc:347] Connected to a master server at 127.28.108.190:39241
I20260807 08:46:05.838433 14388 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:05.838701 14388 heartbeater.cc:511] Master 127.28.108.190:39241 requested a full tablet report, sending...
I20260807 08:46:05.839241 14008 ts_manager.cc:194] Registered new tserver with Master: 87759687187240caa1be334c4e737e48 (127.28.108.130:36607)
I20260807 08:46:05.839854 14008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:51287
W20260807 08:46:05.931906 14393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:05.932183 14393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:05.932214 14393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:05.933529 14393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:05.933591 14393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:05.933650 14393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:46:05.935441 14393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:35531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=44047
--tserver_master_addrs=127.28.108.190:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:05.935721 14393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:05.935930 14393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:05.943290 14401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:05.943253 14403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:05.943313 14400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:05.943985 14393 server_base.cc:1034] running on GCE node
I20260807 08:46:05.944264 14393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:05.944648 14393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:05.945871 14393 hybrid_clock.cc:648] HybridClock initialized: now 1786092365945756 us; error 140 us; skew 500 ppm
Aug 07 08:46:05 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092365, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:05.949183 14393 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:46:05.950189 14393 webserver.cc:460] Webserver started at http://127.28.108.131:44047/ using document root <none> and password file <none>
I20260807 08:46:05.950646 14393 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:05.950744 14393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:05.952822 14393 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:46:05.954008 14410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:05.954532 14393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 08:46:05.954676 14393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "3c66605377eb4e0f97023d5f124a882e"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "488ca65a7729a4a614a7758637ab3011"
server_key_iv: "3af940c186aa838908aacc0c110a6904"
server_key_version: "encryptionkey@0"
I20260807 08:46:05.955240 14393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:05.970347 14393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:05.973302 14393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:05.973450 14393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:05.973814 14393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:05.974195 14393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:05.974259 14393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:05.974339 14393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:05.974375 14393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:05.982074 14393 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:35531
I20260807 08:46:05.982163 14539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:35531 every 8 connection(s)
I20260807 08:46:05.982636 14393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:46:05.990602 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 14393
Aug 07 08:46:05 dist-test-slave-9zdj krb5kdc[13345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092365, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:05.996140 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:05.983962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:58653 (local address 127.28.108.190:39241)
0807 08:46:05.984033 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:05.984035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:05.984361 (+ 326us) server_negotiation.cc:407] Connection header received
0807 08:46:05.984954 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:05.984956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:05.984969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:05.985037 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:05.986171 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.986686 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.987890 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.988099 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.988168 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:05.991739 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:05.991740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:05.991741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:05.991764 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:05.993869 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:05.994564 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:05.994565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:05.994565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:05.994636 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:05.995017 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:05.995018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:05.995018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:05.995179 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:05.995248 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:05.995886 (+ 638us) server_negotiation.cc:299] Negotiation successful
0807 08:46:05.996010 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 08:46:05.996443 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:05.983967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:58653)
0807 08:46:05.984175 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:05.984219 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:05.984791 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:05.985129 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:05.985133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:05.985824 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:05.986060 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:05.986065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.986838 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.986839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:05.987769 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:05.987773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.988217 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.988218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:05.988365 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:05.989157 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:05.989174 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:05.991605 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:05.994025 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:05.994040 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:05.994044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:05.994459 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:05.994748 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:05.994749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:05.994749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:05.994936 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:05.995420 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:05.995430 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:05.995702 (+ 272us) client_negotiation.cc:769] Sending connection context
0807 08:46:05.995854 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 08:46:05.996119 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
I20260807 08:46:05.997095 14540 heartbeater.cc:347] Connected to a master server at 127.28.108.190:39241
I20260807 08:46:05.997397 14540 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:05.998081 14540 heartbeater.cc:511] Master 127.28.108.190:39241 requested a full tablet report, sending...
I20260807 08:46:05.998608 14008 ts_manager.cc:194] Registered new tserver with Master: 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531)
I20260807 08:46:05.999028 14008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:58653
I20260807 08:46:06.002018 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:05.991264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.28.108.190:39241)
0807 08:46:05.991394 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:05.991398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:05.991567 (+ 169us) server_negotiation.cc:407] Connection header received
0807 08:46:05.991621 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:05.991622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:05.991636 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:05.991785 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:05.992794 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.993746 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.994962 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:05.995323 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:05.995393 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:05.997217 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:05.997219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:05.997219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:05.997227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:05.999756 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:06.000416 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:06.000417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:06.000417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:06.000669 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:06.000994 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:06.000996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:06.000996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:06.001152 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:06.001359 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:06.001728 (+ 369us) server_negotiation.cc:299] Negotiation successful
0807 08:46:06.001860 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260807 08:46:06.003175 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:46:06.007328 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.003809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.28.108.190:39241)
0807 08:46:06.003928 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:06.003930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:06.004106 (+ 176us) server_negotiation.cc:407] Connection header received
0807 08:46:06.004279 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:06.004280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:06.004320 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:06.004427 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:46:06.004605 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.005204 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.006201 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.006446 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.006549 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:06.006795 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:46:06.006858 (+ 63us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 08:46:06.007117 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39420: token expired
Metrics: {"server-negotiator.queue_time_us":83}
W20260807 08:46:06.007409 13925 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.108.190:39241)
W20260807 08:46:06.007517 14546 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39420: token expired
I20260807 08:46:06.015885 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.007763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.28.108.190:39241)
0807 08:46:06.007814 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:06.007816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:06.007830 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:46:06.007919 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:06.007920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:06.007929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:06.007976 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:06.008578 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.009118 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.010181 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.010480 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.010548 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:06.012019 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:06.012020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:06.012020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:06.012027 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:06.013890 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:06.014422 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:06.014423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:06.014424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:06.014509 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:06.014862 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:06.014864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:06.014864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:06.015085 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:06.015197 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:06.015595 (+ 398us) server_negotiation.cc:299] Negotiation successful
0807 08:46:06.015724 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260807 08:46:06.017704 14008 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:46:06.019771 14008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:46:06.023029 14156 tablet_service.cc:1467] Processing CreateTablet for tablet f957c1e306ba445e9ae2fab2b234d3bc (DEFAULT_TABLE table=test-table [id=75c7fbb170104defa00ef6a9af847b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:06.023326 14156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f957c1e306ba445e9ae2fab2b234d3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:06.025108 14106 tablet_bootstrap.cc:492] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: Bootstrap starting.
I20260807 08:46:06.025756 14106 tablet_bootstrap.cc:654] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:06.026463 14106 tablet_bootstrap.cc:492] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: No bootstrap required, opened a new log
I20260807 08:46:06.026532 14106 ts_tablet_manager.cc:1397] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 08:46:06.026665 14106 raft_consensus.cc:348] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:06.026765 14106 raft_consensus.cc:374] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:06.026830 14106 raft_consensus.cc:729] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda9e80b479a4129b8b11c335b241d5b, State: Initialized, Role: FOLLOWER
I20260807 08:46:06.026880 14106 consensus_queue.cc:260] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:06.026968 14106 ts_tablet_manager.cc:1428] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 08:46:06.027128 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.022522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:36607 (local address 127.0.0.1:60740)
0807 08:46:06.022609 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:06.022625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:06.022687 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:06.023324 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:06.023326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:06.023328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:06.023391 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:06.023393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.024568 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.024570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:06.025774 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:06.025776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.026878 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.026880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:06.026957 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:06.026962 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:46:06.026993 (+ 31us) client_negotiation.cc:240] Negotiation successful
0807 08:46:06.027054 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 08:46:06.027648 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.022924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.28.108.130:36607)
0807 08:46:06.023116 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:06.023121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:06.023132 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:46:06.023179 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:06.023180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:06.023212 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:06.023314 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:06.023568 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.024467 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.026011 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.026745 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.027368 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:06.027460 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:06.027486 (+ 26us) server_negotiation.cc:299] Negotiation successful
0807 08:46:06.027552 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":116,"threads_started":1}
I20260807 08:46:06.028533 14318 tablet_service.cc:1467] Processing CreateTablet for tablet f957c1e306ba445e9ae2fab2b234d3bc (DEFAULT_TABLE table=test-table [id=75c7fbb170104defa00ef6a9af847b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:06.028956 14318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f957c1e306ba445e9ae2fab2b234d3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:06.029561 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.022522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:35531 (local address 127.0.0.1:47226)
0807 08:46:06.022675 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:06.022686 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:06.022736 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:06.025460 (+ 2724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:06.025461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:06.025464 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:06.025510 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:06.025511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.027148 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.027149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:06.028328 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:06.028332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.029235 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.029236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:06.029318 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:06.029324 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:46:06.029400 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 08:46:06.029462 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":27,"thread_start_us":85,"threads_started":1}
I20260807 08:46:06.029578 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.022925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.28.108.131:35531)
0807 08:46:06.025159 (+ 2234us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:06.025166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:06.025178 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:46:06.025236 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:06.025239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:06.025257 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:06.025347 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:06.025609 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.027017 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.028434 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.029142 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.029209 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:06.029258 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:06.029470 (+ 212us) server_negotiation.cc:299] Negotiation successful
0807 08:46:06.029512 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2165,"thread_start_us":121,"threads_started":1}
I20260807 08:46:06.030789 14464 tablet_service.cc:1467] Processing CreateTablet for tablet f957c1e306ba445e9ae2fab2b234d3bc (DEFAULT_TABLE table=test-table [id=75c7fbb170104defa00ef6a9af847b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:06.031172 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f957c1e306ba445e9ae2fab2b234d3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:06.032776 14556 tablet_bootstrap.cc:492] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: Bootstrap starting.
I20260807 08:46:06.033670 14556 tablet_bootstrap.cc:654] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:06.034309 14556 log.cc:824] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:06.034394 14557 tablet_bootstrap.cc:492] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e: Bootstrap starting.
I20260807 08:46:06.035136 14557 tablet_bootstrap.cc:654] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:06.035444 14556 tablet_bootstrap.cc:492] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: No bootstrap required, opened a new log
I20260807 08:46:06.035540 14556 ts_tablet_manager.cc:1397] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:46:06.035759 14557 log.cc:824] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:06.036875 14557 tablet_bootstrap.cc:492] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e: No bootstrap required, opened a new log
I20260807 08:46:06.036996 14557 ts_tablet_manager.cc:1397] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:46:06.037540 14556 raft_consensus.cc:348] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:06.037786 14556 raft_consensus.cc:374] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:06.037819 14556 raft_consensus.cc:729] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87759687187240caa1be334c4e737e48, State: Initialized, Role: FOLLOWER
I20260807 08:46:06.037981 14556 consensus_queue.cc:260] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:06.038341 14556 ts_tablet_manager.cc:1428] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:46:06.038388 14388 heartbeater.cc:503] Master 127.28.108.190:39241 was elected leader, sending a full tablet report...
I20260807 08:46:06.039175 14557 raft_consensus.cc:348] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:06.039399 14557 raft_consensus.cc:374] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:06.039458 14557 raft_consensus.cc:729] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c66605377eb4e0f97023d5f124a882e, State: Initialized, Role: FOLLOWER
I20260807 08:46:06.039598 14557 consensus_queue.cc:260] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:06.039983 14557 ts_tablet_manager.cc:1428] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:46:06.040028 14540 heartbeater.cc:503] Master 127.28.108.190:39241 was elected leader, sending a full tablet report...
W20260807 08:46:06.074852 14389 tablet.cc:2369] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:46:06.094599 14561 raft_consensus.cc:482] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:06.094820 14561 raft_consensus.cc:504] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:06.095305 14561 leader_election.cc:288] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87759687187240caa1be334c4e737e48 (127.28.108.130:36607), fda9e80b479a4129b8b11c335b241d5b (127.28.108.129:35045)
I20260807 08:46:06.099318 14247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.095762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:40119 (local address 127.28.108.129:35045)
0807 08:46:06.095830 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:06.095832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:06.095852 (+ 20us) server_negotiation.cc:407] Connection header received
0807 08:46:06.095888 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:06.095889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:06.095900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:06.095958 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:06.096157 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.096670 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.098089 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.098795 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.098955 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:06.098968 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:06.099206 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 08:46:06.099251 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 08:46:06.099447 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.095598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:35045 (local address 127.28.108.131:40119)
0807 08:46:06.095658 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:06.095672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:06.095722 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:06.095981 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:06.095983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:06.095985 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:06.096076 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:06.096079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.096750 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.096751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:06.097971 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:06.097975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.098973 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.098974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:06.099059 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:06.099102 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 08:46:06.099188 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 08:46:06.099241 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260807 08:46:06.100430 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.095607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:36607 (local address 127.28.108.131:45969)
0807 08:46:06.095958 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:06.095971 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:06.096071 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:06.096451 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:06.096453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:06.096455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:06.096544 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:06.096545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.097593 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.097594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:06.099023 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:06.099028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.100015 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.100017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:06.100118 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:06.100125 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 08:46:06.100217 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 08:46:06.100265 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":54,"thread_start_us":221,"threads_started":1}
I20260807 08:46:06.100479 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.095830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:45969 (local address 127.28.108.130:36607)
0807 08:46:06.095905 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:06.095907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:06.096299 (+ 392us) server_negotiation.cc:407] Connection header received
0807 08:46:06.096331 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:06.096332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:06.096352 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:06.096429 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:06.096631 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.097440 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.099144 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.099907 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.099980 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:06.099988 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:06.100369 (+ 381us) server_negotiation.cc:299] Negotiation successful
0807 08:46:06.100418 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 08:46:06.099871 14181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f957c1e306ba445e9ae2fab2b234d3bc" candidate_uuid: "3c66605377eb4e0f97023d5f124a882e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fda9e80b479a4129b8b11c335b241d5b" is_pre_election: true
I20260807 08:46:06.100929 14181 raft_consensus.cc:2454] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c66605377eb4e0f97023d5f124a882e in term 0.
I20260807 08:46:06.101248 14414 leader_election.cc:302] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c66605377eb4e0f97023d5f124a882e, fda9e80b479a4129b8b11c335b241d5b; no voters:
I20260807 08:46:06.101351 14340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f957c1e306ba445e9ae2fab2b234d3bc" candidate_uuid: "3c66605377eb4e0f97023d5f124a882e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87759687187240caa1be334c4e737e48" is_pre_election: true
I20260807 08:46:06.101416 14561 raft_consensus.cc:2781] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:46:06.101478 14561 raft_consensus.cc:482] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:46:06.101516 14561 raft_consensus.cc:3037] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:06.101539 14340 raft_consensus.cc:2454] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c66605377eb4e0f97023d5f124a882e in term 0.
I20260807 08:46:06.102641 14561 raft_consensus.cc:504] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:06.102895 14561 leader_election.cc:288] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [CANDIDATE]: Term 1 election: Requested vote from peers 87759687187240caa1be334c4e737e48 (127.28.108.130:36607), fda9e80b479a4129b8b11c335b241d5b (127.28.108.129:35045)
I20260807 08:46:06.103003 14340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f957c1e306ba445e9ae2fab2b234d3bc" candidate_uuid: "3c66605377eb4e0f97023d5f124a882e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87759687187240caa1be334c4e737e48"
I20260807 08:46:06.103048 14181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f957c1e306ba445e9ae2fab2b234d3bc" candidate_uuid: "3c66605377eb4e0f97023d5f124a882e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fda9e80b479a4129b8b11c335b241d5b"
I20260807 08:46:06.103133 14340 raft_consensus.cc:3037] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:06.103178 14181 raft_consensus.cc:3037] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:06.104200 14181 raft_consensus.cc:2454] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c66605377eb4e0f97023d5f124a882e in term 1.
I20260807 08:46:06.104243 14340 raft_consensus.cc:2454] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c66605377eb4e0f97023d5f124a882e in term 1.
I20260807 08:46:06.104455 14413 leader_election.cc:302] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c66605377eb4e0f97023d5f124a882e, 87759687187240caa1be334c4e737e48; no voters:
I20260807 08:46:06.104763 14561 raft_consensus.cc:2781] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:06.105086 14561 raft_consensus.cc:686] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 1 LEADER]: Becoming Leader. State: Replica: 3c66605377eb4e0f97023d5f124a882e, State: Running, Role: LEADER
I20260807 08:46:06.105250 14561 consensus_queue.cc:237] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:06.106560 14008 catalog_manager.cc:5495] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c66605377eb4e0f97023d5f124a882e (127.28.108.131). New cstate: current_term: 1 leader_uuid: "3c66605377eb4e0f97023d5f124a882e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } health_report { overall_health: UNKNOWN } } }
I20260807 08:46:06.129213 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.126211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.28.108.131:35531)
0807 08:46:06.126339 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:06.126341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:06.126358 (+ 17us) server_negotiation.cc:407] Connection header received
0807 08:46:06.126400 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:06.126401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:06.126421 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:06.126484 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:46:06.126703 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.127214 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.128141 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:06.128375 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:06.128456 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:06.128663 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:46:06.128789 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:46:06.128860 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:06.128963 (+ 103us) server_negotiation.cc:299] Negotiation successful
0807 08:46:06.128988 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260807 08:46:06.131534 14181 raft_consensus.cc:1261] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Refusing update from remote peer 3c66605377eb4e0f97023d5f124a882e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:06.131539 14340 raft_consensus.cc:1261] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48 [term 1 FOLLOWER]: Refusing update from remote peer 3c66605377eb4e0f97023d5f124a882e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:06.132066 14563 consensus_queue.cc:1035] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:06.132265 14561 consensus_queue.cc:1035] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [LEADER]: Connected to new peer: Peer: permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:06.136356 14566 mvcc.cc:205] Tried to move back new op lower bound from 7315834331671220224 to 7315834331567775744. Current Snapshot: MvccSnapshot[applied={T|T < 7315834331671220224 or (T in {7315834331671220224})}]
I20260807 08:46:06.138083 14007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:06.138195 14007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:06.139426 14007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Sending DeleteTablet for 3 replicas of tablet f957c1e306ba445e9ae2fab2b234d3bc
I20260807 08:46:06.139854 14318 tablet_service.cc:1514] Processing DeleteTablet for tablet f957c1e306ba445e9ae2fab2b234d3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:06 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:60740
I20260807 08:46:06.139878 14464 tablet_service.cc:1514] Processing DeleteTablet for tablet f957c1e306ba445e9ae2fab2b234d3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:06 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:47226
I20260807 08:46:06.139925 14156 tablet_service.cc:1514] Processing DeleteTablet for tablet f957c1e306ba445e9ae2fab2b234d3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:06 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:34784
I20260807 08:46:06.140049 14248 tablet_replica.cc:330] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: stopping tablet replica
I20260807 08:46:06.140100 14248 raft_consensus.cc:2229] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:06.140146 14248 raft_consensus.cc:2258] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:06.140182 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 14051
I20260807 08:46:06.145283 14578 tablet_replica.cc:330] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: stopping tablet replica
I20260807 08:46:06.145517 14578 raft_consensus.cc:2229] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:06.145714 14578 raft_consensus.cc:2258] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:06.146602 14578 ts_tablet_manager.cc:1905] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:06.148981 14578 ts_tablet_manager.cc:1918] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:06.149111 14578 log.cc:1196] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/f957c1e306ba445e9ae2fab2b234d3bc
I20260807 08:46:06.149501 14578 ts_tablet_manager.cc:1939] T f957c1e306ba445e9ae2fab2b234d3bc P 87759687187240caa1be334c4e737e48: Deleting consensus metadata
I20260807 08:46:06.150223 13992 catalog_manager.cc:4841] TS 87759687187240caa1be334c4e737e48 (127.28.108.130:36607): tablet f957c1e306ba445e9ae2fab2b234d3bc (table test-table [id=75c7fbb170104defa00ef6a9af847b53]) successfully deleted
W20260807 08:46:06.151768 13993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:35045 (error 108)
I20260807 08:46:06.152107 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 14244
I20260807 08:46:06.154235 14577 tablet_replica.cc:330] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e: stopping tablet replica
I20260807 08:46:06.154403 14577 raft_consensus.cc:2229] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:46:06.154512 14577 raft_consensus.cc:2258] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:06.155092 14577 ts_tablet_manager.cc:1905] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:06.157203 14577 ts_tablet_manager.cc:1918] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:06.157320 14577 log.cc:1196] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/f957c1e306ba445e9ae2fab2b234d3bc
I20260807 08:46:06.157838 14577 ts_tablet_manager.cc:1939] T f957c1e306ba445e9ae2fab2b234d3bc P 3c66605377eb4e0f97023d5f124a882e: Deleting consensus metadata
I20260807 08:46:06.158372 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:06.158151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:35045 (local address 127.0.0.1:34790)
0807 08:46:06.158236 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:06.158324 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:35045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 08:46:06.159552 13993 catalog_manager.cc:4584] TS fda9e80b479a4129b8b11c335b241d5b (127.28.108.129:35045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f957c1e306ba445e9ae2fab2b234d3bc: Network error: Client connection negotiation failed: client connection to 127.28.108.129:35045: connect: Connection refused (error 111)
I20260807 08:46:06.160349 13993 catalog_manager.cc:4841] TS 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531): tablet f957c1e306ba445e9ae2fab2b234d3bc (table test-table [id=75c7fbb170104defa00ef6a9af847b53]) successfully deleted
I20260807 08:46:06.160861 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 14393
I20260807 08:46:06.168380 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 13971
I20260807 08:46:26.177990 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:39241
--webserver_interface=127.28.108.190
--webserver_port=37239
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:39241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:26.275307 14581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:26.275583 14581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:26.275610 14581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:26.277033 14581 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 08:46:26.277096 14581 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 08:46:26.277117 14581 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 08:46:26.277129 14581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:26.277143 14581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:26.277156 14581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:26.277205 14581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:26.277245 14581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:26.279182 14581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:39241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:39241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:26.279531 14581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:26.279834 14581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:26.286316 14586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:26.286350 14587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:26.286345 14581 server_base.cc:1034] running on GCE node
W20260807 08:46:26.286419 14590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:26.286917 14581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:26.287240 14581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:26.288416 14581 hybrid_clock.cc:648] HybridClock initialized: now 1786092386288388 us; error 42 us; skew 500 ppm
Aug 07 08:46:26 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092386, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:26.297791 14581 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:46:26.298463 14581 webserver.cc:460] Webserver started at http://127.28.108.190:37239/ using document root <none> and password file <none>
I20260807 08:46:26.298768 14581 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:26.298882 14581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:26.300639 14581 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:46:26.301904 14597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:26.302152 14581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:26.302268 14581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "b5589d728d224773a6a639f0f21bfef6"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "52d627c6b4b3aa41293bc18bbb74d229"
server_key_iv: "e86acc85e3d5e96f251f7a01658bf5ca"
server_key_version: "encryptionkey@0"
I20260807 08:46:26.302711 14581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:26.312023 14581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:26.315095 14581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:26.315244 14581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:26.320405 14581 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:39241
I20260807 08:46:26.320497 14656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:39241 every 8 connection(s)
I20260807 08:46:26.320971 14581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:26.323115 14657 sys_catalog.cc:263] Verifying existing consensus state
I20260807 08:46:26.324048 14657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Bootstrap starting.
I20260807 08:46:26.327137 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 14581
I20260807 08:46:26.327787 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:35045
--local_ip_for_outbound_sockets=127.28.108.129
--tserver_master_addrs=127.28.108.190:39241
--webserver_port=33575
--webserver_interface=127.28.108.129
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:46:26.336830 14657 log.cc:824] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:26.341547 14657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:46:26.342217 14657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Bootstrap complete.
I20260807 08:46:26.346725 14657 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:46:26.347149 14657 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5589d728d224773a6a639f0f21bfef6, State: Initialized, Role: FOLLOWER
I20260807 08:46:26.347359 14657 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:46:26.347437 14657 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:26.347492 14657 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:26.347549 14657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 2 FOLLOWER]: Advancing to term 3
I20260807 08:46:26.349206 14657 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:46:26.349375 14657 leader_election.cc:302] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5589d728d224773a6a639f0f21bfef6; no voters:
I20260807 08:46:26.349588 14657 leader_election.cc:288] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 08:46:26.349748 14661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 08:46:26.349936 14657 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:26.350005 14661 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [term 3 LEADER]: Becoming Leader. State: Replica: b5589d728d224773a6a639f0f21bfef6, State: Running, Role: LEADER
I20260807 08:46:26.350272 14661 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } }
I20260807 08:46:26.350674 14662 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b5589d728d224773a6a639f0f21bfef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } } }
I20260807 08:46:26.350775 14662 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:26.351111 14663 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5589d728d224773a6a639f0f21bfef6. Latest consensus state: current_term: 3 leader_uuid: "b5589d728d224773a6a639f0f21bfef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5589d728d224773a6a639f0f21bfef6" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39241 } } }
I20260807 08:46:26.351310 14663 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:26.351506 14670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:26.352985 14670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:26.353111 14670 catalog_manager.cc:1259] Loaded cluster ID: f99a77caf11d46d99dc12c061ef7d97c
I20260807 08:46:26.353149 14670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:46:26.354393 14670 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:26.354602 14670 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Loaded TSK: 0
I20260807 08:46:26.354671 14670 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5589d728d224773a6a639f0f21bfef6: Loaded TSK: 1
F20260807 08:46:26.355036 14670 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f988902b9ad google::LogMessage::Fail() at ??:0
@ 0x7f988902e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f988902b4de google::LogMessage::Flush() at ??:0
@ 0x7f988902ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f988b798104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f988b795d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9890f8a1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9890f8a48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9890f8a594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f9890f50c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9890f5aad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f988aef90d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f988aeee954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f988bd526db start_thread at ??:0
@ 0x7f9887b2961f clone at ??:0
W20260807 08:46:26.447526 14659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:26.447829 14659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:26.447876 14659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:26.449342 14659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:26.449443 14659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:26.449496 14659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:46:26.451431 14659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:35045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=33575
--tserver_master_addrs=127.28.108.190:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:26.451795 14659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:26.452070 14659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:26.458281 14700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:26.458285 14701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:26.458482 14659 server_base.cc:1034] running on GCE node
W20260807 08:46:26.458285 14704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:26.458958 14659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:26.459414 14659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:26.460642 14659 hybrid_clock.cc:648] HybridClock initialized: now 1786092386460610 us; error 42 us; skew 500 ppm
Aug 07 08:46:26 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092386, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:26.464000 14659 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:46:26.464638 14659 webserver.cc:460] Webserver started at http://127.28.108.129:33575/ using document root <none> and password file <none>
I20260807 08:46:26.464928 14659 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:26.465016 14659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:26.466960 14659 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:26.468036 14711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:26.468364 14659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:26.468470 14659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "fda9e80b479a4129b8b11c335b241d5b"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "7eb2e0c010507a04a67c6ca70398ad25"
server_key_iv: "4df07000902f497787fcad469b6dc065"
server_key_version: "encryptionkey@0"
I20260807 08:46:26.468907 14659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:26.488703 14659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:26.491946 14659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:26.492100 14659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:26.492491 14659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:26.493067 14718 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 08:46:26.495514 14659 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 08:46:26.495586 14659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260807 08:46:26.495627 14659 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 08:46:26.496649 14659 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 08:46:26.496708 14659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260807 08:46:26.496955 14718 tablet_bootstrap.cc:492] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: Bootstrap starting.
I20260807 08:46:26.505745 14659 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:35045
I20260807 08:46:26.505967 14837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:35045 every 8 connection(s)
I20260807 08:46:26.506352 14659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:46:26.507691 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 14659
I20260807 08:46:26.508093 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:36607
--local_ip_for_outbound_sockets=127.28.108.130
--tserver_master_addrs=127.28.108.190:39241
--webserver_port=34063
--webserver_interface=127.28.108.130
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:46:26.511021 14718 log.cc:824] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:26.513844 14718 tablet_bootstrap.cc:492] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 08:46:26.514266 14718 tablet_bootstrap.cc:492] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: Bootstrap complete.
I20260807 08:46:26.515627 14718 ts_tablet_manager.cc:1397] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.014s
I20260807 08:46:26.518059 14718 raft_consensus.cc:348] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:26.518584 14718 raft_consensus.cc:729] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda9e80b479a4129b8b11c335b241d5b, State: Initialized, Role: FOLLOWER
I20260807 08:46:26.518960 14718 consensus_queue.cc:260] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:26.519431 14718 ts_tablet_manager.cc:1428] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260807 08:46:26.606925 14840 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:46:26.507258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:54003)
0807 08:46:26.507608 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:26.507660 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:26.508373 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:26.606687 (+ 98314us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":117,"threads_started":1}
W20260807 08:46:26.607390 14838 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:39241 (0 consecutive failures): Network error: Failed to ping master at 127.28.108.190:39241: Client connection negotiation failed: client connection to 127.28.108.190:39241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 08:46:26.607851 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:26.607687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:50325)
0807 08:46:26.607765 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:26.607822 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260807 08:46:26.608340 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:26.608231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:53127)
0807 08:46:26.608281 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:26.608315 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260807 08:46:26.608529 14838 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 08:46:26.610412 14841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:26.610651 14841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:26.610679 14841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:26.612026 14841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:26.612080 14841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:26.612112 14841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:46:26.613978 14841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:36607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=34063
--tserver_master_addrs=127.28.108.190:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:26.614289 14841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:26.614571 14841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:26.620711 14850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:26.620711 14849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:26.620757 14852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:26.621068 14841 server_base.cc:1034] running on GCE node
I20260807 08:46:26.621407 14841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:26.621773 14841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:26.622987 14841 hybrid_clock.cc:648] HybridClock initialized: now 1786092386622969 us; error 29 us; skew 500 ppm
Aug 07 08:46:26 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092386, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:26.626194 14841 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:46:26.626776 14841 webserver.cc:460] Webserver started at http://127.28.108.130:34063/ using document root <none> and password file <none>
I20260807 08:46:26.627046 14841 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:26.627131 14841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:26.628715 14841 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:26.629535 14861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:26.629739 14841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:26.629864 14841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "87759687187240caa1be334c4e737e48"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "7e98c4cc607e1ee9757c4d94cbc17149"
server_key_iv: "ea415a610174db03de57cbdbc83ad67b"
server_key_version: "encryptionkey@0"
I20260807 08:46:26.630259 14841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:26.662679 14841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:26.665580 14841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:26.665818 14841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:26.666117 14841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:26.666487 14841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:26.666548 14841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:26.666594 14841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:26.666620 14841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:26.673691 14841 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:36607
I20260807 08:46:26.673745 14991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:36607 every 8 connection(s)
I20260807 08:46:26.674228 14841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:46:26.675508 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:26.675073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:38555)
0807 08:46:26.675265 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:26.675352 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
W20260807 08:46:26.676103 14992 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:39241 (0 consecutive failures): Network error: Failed to ping master at 127.28.108.190:39241: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
I20260807 08:46:26.676551 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:26.676406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:33105)
0807 08:46:26.676499 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:26.676530 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 08:46:26.676878 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:26.676803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:60339)
0807 08:46:26.676838 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:26.676862 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260807 08:46:26.677011 14992 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 08:46:26.679620 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 14841
I20260807 08:46:26.679998 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:35531
--local_ip_for_outbound_sockets=127.28.108.131
--tserver_master_addrs=127.28.108.190:39241
--webserver_port=44047
--webserver_interface=127.28.108.131
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:26.772557 14995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:26.772811 14995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:26.772842 14995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:26.774245 14995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:26.774305 14995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:26.774338 14995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:46:26.776059 14995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:35531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=44047
--tserver_master_addrs=127.28.108.190:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:26.776388 14995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:26.776676 14995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:26.782776 15005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:26.782826 15002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:26.782778 15003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:26.783113 14995 server_base.cc:1034] running on GCE node
I20260807 08:46:26.783393 14995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:26.783644 14995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:26.784837 14995 hybrid_clock.cc:648] HybridClock initialized: now 1786092386784835 us; error 47 us; skew 500 ppm
Aug 07 08:46:26 dist-test-slave-9zdj krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092386, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:26.787842 14995 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:46:26.788419 14995 webserver.cc:460] Webserver started at http://127.28.108.131:44047/ using document root <none> and password file <none>
I20260807 08:46:26.788694 14995 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:26.788748 14995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:26.790522 14995 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:46:26.791329 15013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:26.791591 14995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:46:26.791712 14995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "3c66605377eb4e0f97023d5f124a882e"
format_stamp: "Formatted at 2026-08-07 08:45:44 on dist-test-slave-9zdj"
server_key: "488ca65a7729a4a614a7758637ab3011"
server_key_iv: "3af940c186aa838908aacc0c110a6904"
server_key_version: "encryptionkey@0"
I20260807 08:46:26.792155 14995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:26.817006 14995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:26.820343 14995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:26.820580 14995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:26.820883 14995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:26.821285 14995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:26.821353 14995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:26.821389 14995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:26.821419 14995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:26.828770 14995 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:35531
I20260807 08:46:26.828871 15141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:35531 every 8 connection(s)
I20260807 08:46:26.829305 14995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:46:26.830651 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:26.830228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:35671)
0807 08:46:26.830404 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:26.830486 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
W20260807 08:46:26.831183 15142 heartbeater.cc:644] Failed to heartbeat to 127.28.108.190:39241 (0 consecutive failures): Network error: Failed to ping master at 127.28.108.190:39241: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
I20260807 08:46:26.831532 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:26.831419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:47945)
0807 08:46:26.831470 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:26.831506 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260807 08:46:26.831852 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:26.831728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:53253)
0807 08:46:26.831766 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:26.831836 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260807 08:46:26.831982 15142 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 08:46:26.837949 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 14995
I20260807 08:46:27.609433 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:27.609044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:43051)
0807 08:46:27.609241 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:27.609288 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
I20260807 08:46:27.677932 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:27.677545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:58599)
0807 08:46:27.677808 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:27.677867 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":174,"threads_started":1}
I20260807 08:46:27.832670 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:27.832412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:51259)
0807 08:46:27.832566 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:27.832610 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260807 08:46:28.245203 15150 raft_consensus.cc:482] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:28.245365 15150 raft_consensus.cc:504] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:28.245893 15150 leader_election.cc:288] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531), 87759687187240caa1be334c4e737e48 (127.28.108.130:36607)
Aug 07 08:46:28 dist-test-slave-9zdj krb5kdc[13345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092386, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.130@KRBTEST.COM
Aug 07 08:46:28 dist-test-slave-9zdj krb5kdc[13345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092386, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.131@KRBTEST.COM
I20260807 08:46:28.260483 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:28.246231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:38323 (local address 127.28.108.130:36607)
0807 08:46:28.246436 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:28.246449 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:28.246483 (+ 34us) server_negotiation.cc:407] Connection header received
0807 08:46:28.247159 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:28.247163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:28.247252 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:28.247442 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:28.248586 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:28.250210 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:28.252023 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:28.252796 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:28.252936 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:28.256360 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:28.256367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:28.256367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:28.256381 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:28.258496 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:28.259089 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:28.259092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:28.259093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:28.259204 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:28.259500 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:28.259501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:28.259502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:28.259795 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:28.259891 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:28.260223 (+ 332us) server_negotiation.cc:299] Negotiation successful
0807 08:46:28.260356 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":51,"threads_started":1}
I20260807 08:46:28.260468 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:28.246119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:36607 (local address 127.28.108.129:38323)
0807 08:46:28.246366 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:28.246376 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:28.246443 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:28.247542 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:28.247546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:28.248110 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:28.248398 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:28.248407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:28.250396 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:28.250397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:28.251865 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:28.251867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:28.252963 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:28.252964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:28.253125 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:28.253987 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:28.254003 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:28.256074 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:28.258632 (+ 2558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:28.258644 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:28.258647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:28.258984 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:28.259296 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:28.259297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:28.259297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:28.259423 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:28.259881 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:28.259889 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:28.260037 (+ 148us) client_negotiation.cc:769] Sending connection context
0807 08:46:28.260153 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 08:46:28.260356 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":56,"thread_start_us":103,"threads_started":1}
I20260807 08:46:28.260639 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:28.246110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:35531 (local address 127.28.108.129:42257)
0807 08:46:28.246356 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:28.246371 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:28.246443 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:28.247542 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:28.247546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:28.248110 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:28.248398 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:28.248407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:28.250412 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:28.250413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:28.251837 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:28.251843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:28.252963 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:28.252964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:28.253125 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:28.253987 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:28.254000 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:28.256384 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:28.259012 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:28.259013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:28.259014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:28.259249 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:28.259577 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:28.259578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:28.259578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:28.259654 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:28.260180 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:28.260182 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:28.260244 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 08:46:28.260327 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 08:46:28.260459 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":83,"thread_start_us":191,"threads_started":1}
I20260807 08:46:28.260757 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:28.246236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:42257 (local address 127.28.108.131:35531)
0807 08:46:28.246436 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:28.246449 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:28.246484 (+ 35us) server_negotiation.cc:407] Connection header received
0807 08:46:28.247201 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:28.247204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:28.247251 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:28.247442 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:28.248585 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:28.250210 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:28.252011 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:28.252811 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:28.252936 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:28.256525 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:28.256530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:28.256530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:28.256542 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:28.258836 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:28.259377 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:28.259381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:28.259382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:28.259494 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:28.259809 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:28.259810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:28.259811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:28.260086 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:28.260195 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:28.260445 (+ 250us) server_negotiation.cc:299] Negotiation successful
0807 08:46:28.260608 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
W20260807 08:46:28.261704 14715 leader_election.cc:341] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531): Not found: Tablet not found: f957c1e306ba445e9ae2fab2b234d3bc
W20260807 08:46:28.261916 14714 leader_election.cc:341] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87759687187240caa1be334c4e737e48 (127.28.108.130:36607): Not found: Tablet not found: f957c1e306ba445e9ae2fab2b234d3bc
I20260807 08:46:28.261973 14714 leader_election.cc:302] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fda9e80b479a4129b8b11c335b241d5b; no voters: 3c66605377eb4e0f97023d5f124a882e, 87759687187240caa1be334c4e737e48
I20260807 08:46:28.262095 15150 raft_consensus.cc:2726] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:46:28.610369 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:28.610112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:48827)
0807 08:46:28.610266 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:28.610314 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260807 08:46:28.678776 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:28.678443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:54377)
0807 08:46:28.678643 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:28.678710 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":66,"threads_started":1}
I20260807 08:46:28.833945 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:28.833259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:54799)
0807 08:46:28.833472 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:28.833585 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260807 08:46:29.611464 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:29.611140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:45811)
0807 08:46:29.611332 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:29.611408 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260807 08:46:29.679719 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:29.679410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:43749)
0807 08:46:29.679596 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:29.679652 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20260807 08:46:29.834877 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:29.834556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:37207)
0807 08:46:29.834767 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:29.834821 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":149,"threads_started":1}
I20260807 08:46:29.953310 15160 raft_consensus.cc:482] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:29.953485 15160 raft_consensus.cc:504] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:29.953866 15160 leader_election.cc:288] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531), 87759687187240caa1be334c4e737e48 (127.28.108.130:36607)
W20260807 08:46:29.954595 14715 leader_election.cc:341] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531): Not found: Tablet not found: f957c1e306ba445e9ae2fab2b234d3bc
W20260807 08:46:29.954743 14714 leader_election.cc:341] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87759687187240caa1be334c4e737e48 (127.28.108.130:36607): Not found: Tablet not found: f957c1e306ba445e9ae2fab2b234d3bc
I20260807 08:46:29.954854 14714 leader_election.cc:302] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fda9e80b479a4129b8b11c335b241d5b; no voters: 3c66605377eb4e0f97023d5f124a882e, 87759687187240caa1be334c4e737e48
I20260807 08:46:29.955005 15160 raft_consensus.cc:2726] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:46:30.612740 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:30.612433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:43245)
0807 08:46:30.612620 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:30.612684 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20260807 08:46:30.680826 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:30.680454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:52593)
0807 08:46:30.680723 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:30.680774 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
I20260807 08:46:30.835904 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:30.835594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:58733)
0807 08:46:30.835794 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:30.835849 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":110,"threads_started":1}
I20260807 08:46:31.614130 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:31.613744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:35115)
0807 08:46:31.614000 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:31.614062 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":144,"threads_started":1}
I20260807 08:46:31.681881 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:31.681481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:58017)
0807 08:46:31.681777 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:31.681828 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":94,"threads_started":1}
I20260807 08:46:31.836967 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:31.836640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:55451)
0807 08:46:31.836859 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:31.836913 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":118,"threads_started":1}
I20260807 08:46:32.497102 15168 raft_consensus.cc:482] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:32.497323 15168 raft_consensus.cc:504] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:32.497593 15168 leader_election.cc:288] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531), 87759687187240caa1be334c4e737e48 (127.28.108.130:36607)
W20260807 08:46:32.498257 14714 leader_election.cc:341] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87759687187240caa1be334c4e737e48 (127.28.108.130:36607): Not found: Tablet not found: f957c1e306ba445e9ae2fab2b234d3bc
W20260807 08:46:32.498351 14715 leader_election.cc:341] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531): Not found: Tablet not found: f957c1e306ba445e9ae2fab2b234d3bc
I20260807 08:46:32.498385 14714 leader_election.cc:302] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fda9e80b479a4129b8b11c335b241d5b; no voters: 3c66605377eb4e0f97023d5f124a882e, 87759687187240caa1be334c4e737e48
I20260807 08:46:32.498485 15168 raft_consensus.cc:2726] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:46:32.615162 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:32.614877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:42801)
0807 08:46:32.615052 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:32.615107 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
I20260807 08:46:32.682971 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:32.682608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:56731)
0807 08:46:32.682868 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:32.682918 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":161,"threads_started":1}
I20260807 08:46:32.837937 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:32.837659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:58697)
0807 08:46:32.837833 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:32.837881 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260807 08:46:33.616384 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:33.615984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:55875)
0807 08:46:33.616179 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:33.616231 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
I20260807 08:46:33.683979 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:33.683674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:35855)
0807 08:46:33.683824 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:33.683908 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260807 08:46:33.838991 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:33.838616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:45663)
0807 08:46:33.838883 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:33.838935 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
I20260807 08:46:34.617379 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:34.617103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:46051)
0807 08:46:34.617279 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:34.617327 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260807 08:46:34.684952 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:34.684672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:47183)
0807 08:46:34.684847 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:34.684898 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":74,"threads_started":1}
I20260807 08:46:34.839946 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:34.839646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:48937)
0807 08:46:34.839844 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:34.839891 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":130,"threads_started":1}
I20260807 08:46:35.618265 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:35.617980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:54175)
0807 08:46:35.618160 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:35.618210 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260807 08:46:35.686044 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:35.685583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:54259)
0807 08:46:35.685939 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:35.685990 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":258,"threads_started":1}
I20260807 08:46:35.840929 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:35.840599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:56791)
0807 08:46:35.840830 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:35.840874 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260807 08:46:36.350140 15183 raft_consensus.cc:482] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:36.350255 15183 raft_consensus.cc:504] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:36.350432 15183 leader_election.cc:288] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531), 87759687187240caa1be334c4e737e48 (127.28.108.130:36607)
W20260807 08:46:36.350939 14715 leader_election.cc:341] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531): Not found: Tablet not found: f957c1e306ba445e9ae2fab2b234d3bc
W20260807 08:46:36.351038 14714 leader_election.cc:341] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87759687187240caa1be334c4e737e48 (127.28.108.130:36607): Not found: Tablet not found: f957c1e306ba445e9ae2fab2b234d3bc
I20260807 08:46:36.351145 14714 leader_election.cc:302] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fda9e80b479a4129b8b11c335b241d5b; no voters: 3c66605377eb4e0f97023d5f124a882e, 87759687187240caa1be334c4e737e48
I20260807 08:46:36.351420 15183 raft_consensus.cc:2726] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:46:36.619148 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:36.618869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:55591)
0807 08:46:36.619045 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:36.619093 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":104,"threads_started":1}
I20260807 08:46:36.687062 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:36.686756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:57507)
0807 08:46:36.686934 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:36.687011 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260807 08:46:36.842244 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:36.841827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:56017)
0807 08:46:36.842069 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:36.842130 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":101,"threads_started":1}
I20260807 08:46:37.620380 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:37.619913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:53515)
0807 08:46:37.620081 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:37.620288 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260807 08:46:37.688086 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:37.687778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:40869)
0807 08:46:37.687962 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:37.688013 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260807 08:46:37.842967 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:37.842718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:58423)
0807 08:46:37.842870 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:37.842913 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260807 08:46:38.621268 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:38.620991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:59289)
0807 08:46:38.621158 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:38.621205 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20260807 08:46:38.689102 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:38.688840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:46797)
0807 08:46:38.688996 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:38.689047 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260807 08:46:38.844221 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:38.843921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:53767)
0807 08:46:38.844100 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:38.844151 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260807 08:46:39.622296 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:39.622012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:45171)
0807 08:46:39.622172 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:39.622233 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260807 08:46:39.690354 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:39.689981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:60975)
0807 08:46:39.690247 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:39.690299 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":72,"threads_started":1}
I20260807 08:46:39.845196 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:39.844876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:57005)
0807 08:46:39.845084 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:39.845142 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260807 08:46:39.860685 15197 raft_consensus.cc:482] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:39.860839 15197 raft_consensus.cc:504] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c66605377eb4e0f97023d5f124a882e" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 35531 } } peers { permanent_uuid: "87759687187240caa1be334c4e737e48" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 36607 } } peers { permanent_uuid: "fda9e80b479a4129b8b11c335b241d5b" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35045 } }
I20260807 08:46:39.861078 15197 leader_election.cc:288] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531), 87759687187240caa1be334c4e737e48 (127.28.108.130:36607)
W20260807 08:46:39.861572 14714 leader_election.cc:341] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87759687187240caa1be334c4e737e48 (127.28.108.130:36607): Not found: Tablet not found: f957c1e306ba445e9ae2fab2b234d3bc
W20260807 08:46:39.861667 14715 leader_election.cc:341] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c66605377eb4e0f97023d5f124a882e (127.28.108.131:35531): Not found: Tablet not found: f957c1e306ba445e9ae2fab2b234d3bc
I20260807 08:46:39.861709 14714 leader_election.cc:302] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fda9e80b479a4129b8b11c335b241d5b; no voters: 3c66605377eb4e0f97023d5f124a882e, 87759687187240caa1be334c4e737e48
I20260807 08:46:39.861786 15197 raft_consensus.cc:2726] T f957c1e306ba445e9ae2fab2b234d3bc P fda9e80b479a4129b8b11c335b241d5b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 08:46:40.623289 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:40.622992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:50167)
0807 08:46:40.623181 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:40.623236 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
I20260807 08:46:40.691463 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:40.691123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:60887)
0807 08:46:40.691356 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:40.691408 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":157,"threads_started":1}
I20260807 08:46:40.846338 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:40.846055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:60367)
0807 08:46:40.846226 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:40.846281 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260807 08:46:41.624334 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:41.624049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.129:47363)
0807 08:46:41.624217 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:41.624275 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260807 08:46:41.692698 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:41.692453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.130:41973)
0807 08:46:41.692599 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:41.692646 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260807 08:46:41.845374 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 14659
I20260807 08:46:41.849000 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:41.847818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39241 (local address 127.28.108.131:33757)
0807 08:46:41.848879 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:41.848933 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.190:39241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1029,"thread_start_us":83,"threads_started":1}
I20260807 08:46:41.854890 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 14841
I20260807 08:46:41.862871 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 14995
2026-08-07T08:46:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58991 ms)
[----------] 30 tests from SecurityITest (268334 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:46:41 dist-test-slave-9zdj krb5kdc[15206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:46:41 dist-test-slave-9zdj krb5kdc[15206](info): set up 2 sockets
Aug 07 08:46:41 dist-test-slave-9zdj krb5kdc[15206](info): commencing operation
krb5kdc: starting...
W20260807 08:46:43.092258 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:46:43 dist-test-slave-9zdj krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:46:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:46:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:46:43.245551 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:41905
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:41905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:43.336126 15219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:43.336433 15219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:43.336474 15219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:43.337925 15219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:43.338022 15219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:43.338047 15219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:43.338068 15219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:43.338102 15219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:43.339807 15219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:41905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:41905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:43.340134 15219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:43.340404 15219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:43.345904 15225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:43.345894 15226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:43.346105 15219 server_base.cc:1034] running on GCE node
W20260807 08:46:43.345897 15229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:43.346567 15219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:43.346927 15219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:43.348140 15219 hybrid_clock.cc:648] HybridClock initialized: now 1786092403348114 us; error 42 us; skew 500 ppm
Aug 07 08:46:43 dist-test-slave-9zdj krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092403, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:43.351328 15219 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:46:43.351846 15219 webserver.cc:460] Webserver started at http://127.28.108.190:42517/ using document root <none> and password file <none>
I20260807 08:46:43.352121 15219 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:43.352192 15219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:43.352449 15219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:43.353796 15219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "536859dd19aa41c4888a9898fd6ffc11"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "1c648c95388afcb1d43bd225e7761f4c"
server_key_iv: "68884cd9d86fd807f62ce5b45d9820fc"
server_key_version: "encryptionkey@0"
I20260807 08:46:43.354197 15219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "536859dd19aa41c4888a9898fd6ffc11"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "1c648c95388afcb1d43bd225e7761f4c"
server_key_iv: "68884cd9d86fd807f62ce5b45d9820fc"
server_key_version: "encryptionkey@0"
I20260807 08:46:43.355715 15219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:46:43.356637 15236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:43.356930 15219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:43.357059 15219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "536859dd19aa41c4888a9898fd6ffc11"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "1c648c95388afcb1d43bd225e7761f4c"
server_key_iv: "68884cd9d86fd807f62ce5b45d9820fc"
server_key_version: "encryptionkey@0"
I20260807 08:46:43.357147 15219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:43.392849 15219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:43.395394 15219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:43.395608 15219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:43.400870 15219 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:41905
I20260807 08:46:43.400915 15291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:41905 every 8 connection(s)
I20260807 08:46:43.401343 15219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:43.402415 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:43.403620 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 15219
I20260807 08:46:43.403841 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:46:43.404062 29106 external_mini_cluster.cc:1428] Setting key 364ea6bf12a0d69bfe11f80fcd5c3566
I20260807 08:46:43.406147 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11: Bootstrap starting.
I20260807 08:46:43.406942 15292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:43.407377 15292 log.cc:824] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:43.408353 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11: No bootstrap required, opened a new log
I20260807 08:46:43.411039 15292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "536859dd19aa41c4888a9898fd6ffc11" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41905 } }
I20260807 08:46:43.411279 15292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:43.411340 15292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536859dd19aa41c4888a9898fd6ffc11, State: Initialized, Role: FOLLOWER
I20260807 08:46:43.411514 15292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "536859dd19aa41c4888a9898fd6ffc11" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41905 } }
I20260807 08:46:43.411623 15292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:43.411666 15292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:43.411720 15292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:43.412463 15292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "536859dd19aa41c4888a9898fd6ffc11" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41905 } }
I20260807 08:46:43.412621 15292 leader_election.cc:302] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 536859dd19aa41c4888a9898fd6ffc11; no voters:
I20260807 08:46:43.412861 15292 leader_election.cc:288] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:46:43.412974 15297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:43.413189 15292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:43.413244 15297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [term 1 LEADER]: Becoming Leader. State: Replica: 536859dd19aa41c4888a9898fd6ffc11, State: Running, Role: LEADER
Aug 07 08:46:43 dist-test-slave-9zdj krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:43.413470 15297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "536859dd19aa41c4888a9898fd6ffc11" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41905 } }
I20260807 08:46:43.413926 15298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "536859dd19aa41c4888a9898fd6ffc11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "536859dd19aa41c4888a9898fd6ffc11" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41905 } } }
I20260807 08:46:43.414013 15298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:43.414459 15299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 536859dd19aa41c4888a9898fd6ffc11. Latest consensus state: current_term: 1 leader_uuid: "536859dd19aa41c4888a9898fd6ffc11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "536859dd19aa41c4888a9898fd6ffc11" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41905 } } }
I20260807 08:46:43.414525 15299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11 [sys.catalog]: This master's current role is: LEADER
W20260807 08:46:43.415627 15317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:46:43.415733 15317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:46:43.415838 15305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:43.416034 15305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:43.417876 15305 catalog_manager.cc:1347] Generated new cluster ID: 14272b08844143b2a2f3523ac374823a
I20260807 08:46:43.417936 15305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:46:43.419286 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:43.404881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.28.108.190:41905)
0807 08:46:43.405164 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:43.405177 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:43.405218 (+ 41us) server_negotiation.cc:407] Connection header received
0807 08:46:43.405863 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:43.405875 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:43.405949 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:43.406156 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:43.407056 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:43.408599 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:43.409681 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:43.410505 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:43.410661 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:43.414150 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:43.414156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:43.414157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:43.414175 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:43.416938 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:43.417643 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:43.417647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:43.417648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:43.417831 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:43.418121 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:43.418123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:43.418123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:43.418466 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:43.418613 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:43.418850 (+ 237us) server_negotiation.cc:299] Negotiation successful
0807 08:46:43.419028 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":121,"threads_started":1}
I20260807 08:46:43.430375 15305 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:46:43.431051 15305 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:43.444782 15305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11: Generated new TSK 0
I20260807 08:46:43.445046 15305 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:46:43.490587 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41905
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:43.591431 15322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:43.591701 15322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:43.591729 15322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:43.593039 15322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:43.593094 15322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:43.593125 15322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:46:43.594857 15322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:43.595108 15322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:43.595335 15322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:43.601744 15331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:43.601718 15333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:43.601832 15322 server_base.cc:1034] running on GCE node
W20260807 08:46:43.601718 15330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:43.602419 15322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:43.602749 15322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:43.603981 15322 hybrid_clock.cc:648] HybridClock initialized: now 1786092403603962 us; error 37 us; skew 500 ppm
Aug 07 08:46:43 dist-test-slave-9zdj krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092403, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:43.607815 15322 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:46:43.608551 15322 webserver.cc:460] Webserver started at http://127.28.108.129:42301/ using document root <none> and password file <none>
I20260807 08:46:43.608906 15322 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:43.608994 15322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:43.609190 15322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:43.610522 15322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "bb44718abfae4e629b47a910e819b6d6"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "a03cf7986b12b1635f77489f314d1928"
server_key_iv: "c6504b7d540d8ba37f92a58d6a4e739f"
server_key_version: "encryptionkey@0"
I20260807 08:46:43.611030 15322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "bb44718abfae4e629b47a910e819b6d6"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "a03cf7986b12b1635f77489f314d1928"
server_key_iv: "c6504b7d540d8ba37f92a58d6a4e739f"
server_key_version: "encryptionkey@0"
I20260807 08:46:43.612879 15322 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260807 08:46:43.614058 15341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:43.614346 15322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:43.614463 15322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "bb44718abfae4e629b47a910e819b6d6"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "a03cf7986b12b1635f77489f314d1928"
server_key_iv: "c6504b7d540d8ba37f92a58d6a4e739f"
server_key_version: "encryptionkey@0"
I20260807 08:46:43.614537 15322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:43.631420 15322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:43.634171 15322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:43.634373 15322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:43.634645 15322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:43.635058 15322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:43.635097 15322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:43.635123 15322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:43.635138 15322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:43.642781 15322 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:35585
I20260807 08:46:43.642825 15470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:35585 every 8 connection(s)
I20260807 08:46:43.643231 15322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:46:43.648469 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 15322
I20260807 08:46:43.648597 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:46:43.648835 29106 external_mini_cluster.cc:1428] Setting key 8a16ddb241389b49755d62b51b673302
Aug 07 08:46:43 dist-test-slave-9zdj krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092403, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:43.656777 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:43.644099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:58843 (local address 127.28.108.190:41905)
0807 08:46:43.644152 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:43.644155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:43.644448 (+ 293us) server_negotiation.cc:407] Connection header received
0807 08:46:43.645052 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:43.645054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:43.645069 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:43.645157 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:43.647166 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:43.647925 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:43.649701 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:43.650082 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:43.650176 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:43.652906 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:43.652908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:43.652909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:43.652918 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:43.655035 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:43.655541 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:43.655542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:43.655543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:43.655624 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:43.655935 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:43.655938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:43.655938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:43.656129 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:43.656211 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:43.656587 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 08:46:43.656688 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:46:43.657187 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:43.644084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41905 (local address 127.28.108.129:58843)
0807 08:46:43.644297 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:43.644341 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:43.644925 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:43.645280 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:43.645285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:43.646631 (+ 1346us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:43.646998 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:43.647006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:43.648086 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:43.648087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:43.649544 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:43.649548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:43.650205 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:43.650206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:43.650364 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:43.650992 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:43.651003 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:43.652770 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:43.655157 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:43.655168 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:43.655170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:43.655452 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:43.655717 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:43.655718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:43.655718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:43.655856 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:43.656212 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:43.656219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:43.656424 (+ 205us) client_negotiation.cc:769] Sending connection context
0807 08:46:43.656563 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 08:46:43.656865 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":105,"threads_started":1}
I20260807 08:46:43.657827 15471 heartbeater.cc:347] Connected to a master server at 127.28.108.190:41905
I20260807 08:46:43.657984 15471 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:43.658210 15471 heartbeater.cc:511] Master 127.28.108.190:41905 requested a full tablet report, sending...
I20260807 08:46:43.658877 15254 ts_manager.cc:194] Registered new tserver with Master: bb44718abfae4e629b47a910e819b6d6 (127.28.108.129:35585)
I20260807 08:46:43.659755 15254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:58843
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:46:43.686949 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41905
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:43.787215 15476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:43.787480 15476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:43.787511 15476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:43.788838 15476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:43.788880 15476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:43.788910 15476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:46:43.790740 15476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:43.790999 15476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:43.791234 15476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:43.797226 15486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:43.797226 15484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:43.797427 15483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:43.797582 15476 server_base.cc:1034] running on GCE node
I20260807 08:46:43.797891 15476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:43.798141 15476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:43.799510 15476 hybrid_clock.cc:648] HybridClock initialized: now 1786092403799485 us; error 37 us; skew 500 ppm
Aug 07 08:46:43 dist-test-slave-9zdj krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092403, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:43.802930 15476 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:46:43.803593 15476 webserver.cc:460] Webserver started at http://127.28.108.130:41981/ using document root <none> and password file <none>
I20260807 08:46:43.803907 15476 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:43.803994 15476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:43.804351 15476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:43.805760 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "f62c41c9f7ca4279a430e6387349294e"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "04350ef1917f3755d239f9e38456a03c"
server_key_iv: "74fe7d4ebb7f5300cea7dd7447859b8a"
server_key_version: "encryptionkey@0"
I20260807 08:46:43.806193 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "f62c41c9f7ca4279a430e6387349294e"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "04350ef1917f3755d239f9e38456a03c"
server_key_iv: "74fe7d4ebb7f5300cea7dd7447859b8a"
server_key_version: "encryptionkey@0"
I20260807 08:46:43.807911 15476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:46:43.809051 15494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:43.809477 15476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:43.809834 15476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "f62c41c9f7ca4279a430e6387349294e"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "04350ef1917f3755d239f9e38456a03c"
server_key_iv: "74fe7d4ebb7f5300cea7dd7447859b8a"
server_key_version: "encryptionkey@0"
I20260807 08:46:43.809957 15476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:43.823560 15476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:43.826685 15476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:43.826997 15476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:43.827353 15476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:43.827770 15476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:43.827837 15476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:43.827879 15476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:43.827915 15476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:43.835304 15476 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:45729
I20260807 08:46:43.835364 15621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:45729 every 8 connection(s)
I20260807 08:46:43.835784 15476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
Aug 07 08:46:43 dist-test-slave-9zdj krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092403, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:43.845444 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 15476
I20260807 08:46:43.845567 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:46:43.845842 29106 external_mini_cluster.cc:1428] Setting key 2e1f24dbbb551d7ff813d3c9ae7c8a16
I20260807 08:46:43.848254 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:43.836859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:60221 (local address 127.28.108.190:41905)
0807 08:46:43.836945 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:43.836948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:43.837396 (+ 448us) server_negotiation.cc:407] Connection header received
0807 08:46:43.838199 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:43.838201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:43.838215 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:43.838353 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:43.839586 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:43.840107 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:43.841431 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:43.841858 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:43.841935 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:43.844672 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:43.844674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:43.844674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:43.844686 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:43.846648 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:43.847119 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:43.847121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:43.847121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:43.847197 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:43.847499 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:43.847500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:43.847501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:43.847635 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:43.847728 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:43.848057 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 08:46:43.848160 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 08:46:43.848553 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:43.836829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41905 (local address 127.28.108.130:60221)
0807 08:46:43.837128 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:43.837279 (+ 151us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:43.838039 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:43.838390 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:43.838395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:43.839074 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:43.839444 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:43.839450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:43.840350 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:43.840351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:43.841274 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:43.841277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:43.841965 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:43.841966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:43.842103 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:43.842793 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:43.842804 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:43.844526 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:43.846732 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:43.846743 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:43.846746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:43.847018 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:43.847279 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:43.847280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:43.847281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:43.847418 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:43.847725 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:43.847735 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:43.847900 (+ 165us) client_negotiation.cc:769] Sending connection context
0807 08:46:43.848018 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 08:46:43.848206 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":121,"threads_started":1}
I20260807 08:46:43.849196 15623 heartbeater.cc:347] Connected to a master server at 127.28.108.190:41905
I20260807 08:46:43.849334 15623 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:43.849555 15623 heartbeater.cc:511] Master 127.28.108.190:41905 requested a full tablet report, sending...
I20260807 08:46:43.850063 15254 ts_manager.cc:194] Registered new tserver with Master: f62c41c9f7ca4279a430e6387349294e (127.28.108.130:45729)
I20260807 08:46:43.850622 15254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:60221
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:46:43.882534 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41905
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:43.980908 15628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:43.981181 15628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:43.981220 15628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:43.982878 15628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:43.982951 15628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:43.982986 15628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:46:43.984846 15628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:43.985249 15628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:43.985563 15628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:43.992086 15633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:43.992051 15635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:43.992051 15637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:43.992517 15628 server_base.cc:1034] running on GCE node
I20260807 08:46:43.992775 15628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:43.993067 15628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:43.994366 15628 hybrid_clock.cc:648] HybridClock initialized: now 1786092403994344 us; error 51 us; skew 500 ppm
Aug 07 08:46:43 dist-test-slave-9zdj krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092403, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:43.997606 15628 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:46:43.998179 15628 webserver.cc:460] Webserver started at http://127.28.108.131:37383/ using document root <none> and password file <none>
I20260807 08:46:43.998467 15628 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:43.998544 15628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:43.998734 15628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:43.999907 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "f6bb0162daf64b29bc3fbd6ad120ac01"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "1e60904180a10cd57a0c9c713f7f69c0"
server_key_iv: "adce56ea25f311c963e94c541ab82255"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.000479 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "f6bb0162daf64b29bc3fbd6ad120ac01"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "1e60904180a10cd57a0c9c713f7f69c0"
server_key_iv: "adce56ea25f311c963e94c541ab82255"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.002087 15628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 08:46:44.003000 15645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:44.003417 15628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:44.003587 15628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "f6bb0162daf64b29bc3fbd6ad120ac01"
format_stamp: "Formatted at 2026-08-07 08:46:43 on dist-test-slave-9zdj"
server_key: "1e60904180a10cd57a0c9c713f7f69c0"
server_key_iv: "adce56ea25f311c963e94c541ab82255"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.003727 15628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:44.033893 15628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:44.036763 15628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:44.036973 15628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:44.037277 15628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:44.037712 15628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:44.037782 15628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:44.037842 15628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:44.037875 15628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:44.045087 15628 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:44387
I20260807 08:46:44.045189 15774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:44387 every 8 connection(s)
I20260807 08:46:44.045588 15628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:46:44.050246 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 15628
I20260807 08:46:44.050384 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:46:44.050561 29106 external_mini_cluster.cc:1428] Setting key 344aba6baa8b26ff5026b65b155543ea
Aug 07 08:46:44 dist-test-slave-9zdj krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092403, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:44.057372 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.046560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:52069 (local address 127.28.108.190:41905)
0807 08:46:44.046673 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:44.046676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:44.046926 (+ 250us) server_negotiation.cc:407] Connection header received
0807 08:46:44.047529 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:44.047531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:44.047545 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:44.047608 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:44.048727 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.049387 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.050910 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.051226 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.051311 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.054143 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:44.054145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:44.054145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:44.054153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:44.055891 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:44.056431 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:44.056432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:44.056432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:44.056491 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:44.056755 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:44.056755 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:44.056755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:44.056881 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:44.056936 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:44.057215 (+ 279us) server_negotiation.cc:299] Negotiation successful
0807 08:46:44.057293 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 08:46:44.057785 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.046455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41905 (local address 127.28.108.131:52069)
0807 08:46:44.046712 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:44.046821 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:44.047408 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:44.047704 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:44.047708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:44.048368 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:44.048596 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.048602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.049510 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.049511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.050833 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.050837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.051386 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.051387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.051512 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.052197 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:44.052207 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:44.054021 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:44.056001 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:44.056011 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:44.056014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:44.056345 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:44.056562 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:44.056563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:44.056563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:44.056695 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:44.056945 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:44.056950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:44.057091 (+ 141us) client_negotiation.cc:769] Sending connection context
0807 08:46:44.057191 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 08:46:44.057342 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
I20260807 08:46:44.058480 15775 heartbeater.cc:347] Connected to a master server at 127.28.108.190:41905
I20260807 08:46:44.058580 15775 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:44.058748 15775 heartbeater.cc:511] Master 127.28.108.190:41905 requested a full tablet report, sending...
I20260807 08:46:44.059171 15254 ts_manager.cc:194] Registered new tserver with Master: f6bb0162daf64b29bc3fbd6ad120ac01 (127.28.108.131:44387)
I20260807 08:46:44.059684 15254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:52069
I20260807 08:46:44.063104 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:46:44.072387 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.064609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.28.108.190:41905)
0807 08:46:44.064701 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:44.064703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:44.064778 (+ 75us) server_negotiation.cc:407] Connection header received
0807 08:46:44.064840 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:44.064841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:44.064853 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:44.064910 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:44.065781 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.066485 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.067348 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.067553 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.067624 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.069097 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:44.069098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:44.069099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:44.069107 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:44.070908 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:44.071327 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:44.071328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:44.071329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:44.071398 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:44.071628 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:44.071630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:44.071630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:44.071835 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:44.071935 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:44.072056 (+ 121us) server_negotiation.cc:299] Negotiation successful
0807 08:46:44.072137 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 08:46:44.074589 15254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:46:44.076655 15254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:46:44.084193 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.079565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:44387 (local address 127.0.0.1:50024)
0807 08:46:44.079815 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:44.079858 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:44.079939 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:44.080384 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:44.080388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:44.080391 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:44.080481 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.080484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.081659 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.081660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.082772 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.082775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.083825 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.083826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.083915 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.083931 (+ 16us) client_negotiation.cc:769] Sending connection context
0807 08:46:44.084019 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 08:46:44.084099 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":125,"threads_started":1}
I20260807 08:46:44.084214 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.079709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.28.108.131:44387)
0807 08:46:44.080121 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:44.080128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:44.080140 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:46:44.080190 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:44.080193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:44.080211 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:44.080333 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:44.080585 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.081516 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.082853 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.083710 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.083812 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.083889 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:44.084050 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 08:46:44.084099 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":102,"threads_started":1}
I20260807 08:46:44.084383 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.079559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:45729 (local address 127.0.0.1:58018)
0807 08:46:44.079810 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:44.079858 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:44.079940 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:44.080860 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:44.080861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:44.080864 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:44.080923 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.080924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.082430 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.082431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.083295 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.083297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.084108 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.084109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.084169 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.084173 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:46:44.084234 (+ 61us) client_negotiation.cc:240] Negotiation successful
0807 08:46:44.084279 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":2,"thread_start_us":190,"threads_started":1}
I20260807 08:46:44.085016 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.079928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.28.108.130:45729)
0807 08:46:44.080673 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:44.080679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:44.080690 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:46:44.080731 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:44.080733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:44.080749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:44.081009 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:44.081087 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.082311 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.083399 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.084013 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.084514 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.084587 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:44.084616 (+ 29us) server_negotiation.cc:299] Negotiation successful
0807 08:46:44.084662 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":97,"threads_started":1}
I20260807 08:46:44.085348 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.079887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:35585 (local address 127.0.0.1:52934)
0807 08:46:44.080459 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:44.080471 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:44.080538 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:44.080801 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:44.080801 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:44.080804 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:44.080857 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.080859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.082354 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.082355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.083708 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.083711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.084699 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.084700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.084770 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.084805 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 08:46:44.084857 (+ 52us) client_negotiation.cc:240] Negotiation successful
0807 08:46:44.085269 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":30,"thread_start_us":77,"threads_started":1}
I20260807 08:46:44.085784 15700 tablet_service.cc:1467] Processing CreateTablet for tablet 62e22bbecef14afd83b9bd4aadcc2e17 (DEFAULT_TABLE table=test-table [id=83b6b0b480f249d296e747ca2bf27e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:44.085821 15546 tablet_service.cc:1467] Processing CreateTablet for tablet 62e22bbecef14afd83b9bd4aadcc2e17 (DEFAULT_TABLE table=test-table [id=83b6b0b480f249d296e747ca2bf27e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:44.086184 15546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e22bbecef14afd83b9bd4aadcc2e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:44.086211 15700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e22bbecef14afd83b9bd4aadcc2e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:44.089373 15793 tablet_bootstrap.cc:492] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: Bootstrap starting.
I20260807 08:46:44.089790 15794 tablet_bootstrap.cc:492] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: Bootstrap starting.
I20260807 08:46:44.089998 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.080216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.28.108.129:35585)
0807 08:46:44.080489 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:44.080494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:44.080623 (+ 129us) server_negotiation.cc:407] Connection header received
0807 08:46:44.080677 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:44.080680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:44.080699 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:44.080803 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:44.080929 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.082215 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.083843 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.084588 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.084701 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.089734 (+ 5033us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:44.089800 (+ 66us) server_negotiation.cc:299] Negotiation successful
0807 08:46:44.089888 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":129,"threads_started":1}
I20260807 08:46:44.090378 15793 tablet_bootstrap.cc:654] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:44.090582 15794 tablet_bootstrap.cc:654] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:44.090873 15393 tablet_service.cc:1467] Processing CreateTablet for tablet 62e22bbecef14afd83b9bd4aadcc2e17 (DEFAULT_TABLE table=test-table [id=83b6b0b480f249d296e747ca2bf27e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:44.091114 15794 log.cc:824] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:44.091147 15793 log.cc:824] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:44.091321 15393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e22bbecef14afd83b9bd4aadcc2e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:44.092176 15794 tablet_bootstrap.cc:492] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: No bootstrap required, opened a new log
I20260807 08:46:44.092168 15793 tablet_bootstrap.cc:492] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: No bootstrap required, opened a new log
I20260807 08:46:44.092239 15793 ts_tablet_manager.cc:1397] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:46:44.092238 15794 ts_tablet_manager.cc:1397] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:46:44.094580 15797 tablet_bootstrap.cc:492] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6: Bootstrap starting.
I20260807 08:46:44.094548 15793 raft_consensus.cc:348] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 44387 } } peers { permanent_uuid: "f62c41c9f7ca4279a430e6387349294e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 45729 } } peers { permanent_uuid: "bb44718abfae4e629b47a910e819b6d6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35585 } }
I20260807 08:46:44.094744 15793 raft_consensus.cc:374] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:44.094548 15794 raft_consensus.cc:348] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 44387 } } peers { permanent_uuid: "f62c41c9f7ca4279a430e6387349294e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 45729 } } peers { permanent_uuid: "bb44718abfae4e629b47a910e819b6d6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35585 } }
I20260807 08:46:44.094813 15793 raft_consensus.cc:729] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62c41c9f7ca4279a430e6387349294e, State: Initialized, Role: FOLLOWER
I20260807 08:46:44.094843 15794 raft_consensus.cc:374] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:44.094882 15794 raft_consensus.cc:729] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6bb0162daf64b29bc3fbd6ad120ac01, State: Initialized, Role: FOLLOWER
I20260807 08:46:44.095012 15794 consensus_queue.cc:260] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 44387 } } peers { permanent_uuid: "f62c41c9f7ca4279a430e6387349294e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 45729 } } peers { permanent_uuid: "bb44718abfae4e629b47a910e819b6d6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35585 } }
I20260807 08:46:44.095046 15793 consensus_queue.cc:260] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 44387 } } peers { permanent_uuid: "f62c41c9f7ca4279a430e6387349294e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 45729 } } peers { permanent_uuid: "bb44718abfae4e629b47a910e819b6d6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35585 } }
I20260807 08:46:44.095396 15794 ts_tablet_manager.cc:1428] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 08:46:44.095408 15793 ts_tablet_manager.cc:1428] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260807 08:46:44.095402 15797 tablet_bootstrap.cc:654] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:44.095623 15775 heartbeater.cc:503] Master 127.28.108.190:41905 was elected leader, sending a full tablet report...
I20260807 08:46:44.095642 15623 heartbeater.cc:503] Master 127.28.108.190:41905 was elected leader, sending a full tablet report...
I20260807 08:46:44.096172 15797 log.cc:824] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:44.097276 15797 tablet_bootstrap.cc:492] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6: No bootstrap required, opened a new log
I20260807 08:46:44.097347 15797 ts_tablet_manager.cc:1397] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:46:44.099050 15797 raft_consensus.cc:348] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 44387 } } peers { permanent_uuid: "f62c41c9f7ca4279a430e6387349294e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 45729 } } peers { permanent_uuid: "bb44718abfae4e629b47a910e819b6d6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35585 } }
I20260807 08:46:44.099223 15797 raft_consensus.cc:374] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:44.099265 15797 raft_consensus.cc:729] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb44718abfae4e629b47a910e819b6d6, State: Initialized, Role: FOLLOWER
I20260807 08:46:44.099438 15797 consensus_queue.cc:260] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 44387 } } peers { permanent_uuid: "f62c41c9f7ca4279a430e6387349294e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 45729 } } peers { permanent_uuid: "bb44718abfae4e629b47a910e819b6d6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35585 } }
I20260807 08:46:44.099808 15797 ts_tablet_manager.cc:1428] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 08:46:44.099833 15471 heartbeater.cc:503] Master 127.28.108.190:41905 was elected leader, sending a full tablet report...
W20260807 08:46:44.144181 15472 tablet.cc:2369] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:46:44.296478 15776 tablet.cc:2369] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:46:44.329838 15801 raft_consensus.cc:482] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:44.329998 15801 raft_consensus.cc:504] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 44387 } } peers { permanent_uuid: "f62c41c9f7ca4279a430e6387349294e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 45729 } } peers { permanent_uuid: "bb44718abfae4e629b47a910e819b6d6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35585 } }
I20260807 08:46:44.330669 15801 leader_election.cc:288] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6bb0162daf64b29bc3fbd6ad120ac01 (127.28.108.131:44387), f62c41c9f7ca4279a430e6387349294e (127.28.108.130:45729)
I20260807 08:46:44.334398 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.331043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:41585 (local address 127.28.108.131:44387)
0807 08:46:44.331101 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:44.331103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:44.331218 (+ 115us) server_negotiation.cc:407] Connection header received
0807 08:46:44.331334 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:44.331335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:44.331355 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:44.331408 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:44.331621 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.332165 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.333414 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.333968 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.334068 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.334080 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:44.334270 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 08:46:44.334315 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 08:46:44.334409 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.331115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:49421 (local address 127.28.108.130:45729)
0807 08:46:44.331177 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:44.331179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:44.331227 (+ 48us) server_negotiation.cc:407] Connection header received
0807 08:46:44.331342 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:44.331344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:44.331359 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:44.331432 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:44.331641 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.332165 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.333497 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.333977 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.334089 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.334099 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:44.334287 (+ 188us) server_negotiation.cc:299] Negotiation successful
0807 08:46:44.334332 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 08:46:44.334436 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.330976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:45729 (local address 127.28.108.129:49421)
0807 08:46:44.331154 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:44.331167 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:44.331246 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:44.331437 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:44.331439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:44.331441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:44.331548 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.331549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.332274 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.332275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.333378 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.333382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.334090 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.334092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.334186 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.334190 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:46:44.334265 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 08:46:44.334318 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260807 08:46:44.334450 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.330901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:44387 (local address 127.28.108.129:41585)
0807 08:46:44.331093 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:44.331109 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:44.331237 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:44.331423 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:44.331424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:44.331426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:44.331522 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.331524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.332282 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.332283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.333288 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.333291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.334069 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.334070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.334162 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.334173 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 08:46:44.334249 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 08:46:44.334305 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":54,"thread_start_us":110,"threads_started":1}
I20260807 08:46:44.334928 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e22bbecef14afd83b9bd4aadcc2e17" candidate_uuid: "bb44718abfae4e629b47a910e819b6d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62c41c9f7ca4279a430e6387349294e" is_pre_election: true
I20260807 08:46:44.335143 15570 raft_consensus.cc:2454] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb44718abfae4e629b47a910e819b6d6 in term 0.
I20260807 08:46:44.335151 15723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e22bbecef14afd83b9bd4aadcc2e17" candidate_uuid: "bb44718abfae4e629b47a910e819b6d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" is_pre_election: true
I20260807 08:46:44.335368 15723 raft_consensus.cc:2454] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb44718abfae4e629b47a910e819b6d6 in term 0.
I20260807 08:46:44.335510 15342 leader_election.cc:302] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb44718abfae4e629b47a910e819b6d6, f62c41c9f7ca4279a430e6387349294e; no voters:
I20260807 08:46:44.335736 15801 raft_consensus.cc:2781] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:46:44.335786 15801 raft_consensus.cc:482] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:46:44.335804 15801 raft_consensus.cc:3037] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [term 0 FOLLOWER]: Advancing to term 1
W20260807 08:46:44.336565 15624 tablet.cc:2369] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:46:44.336789 15801 raft_consensus.cc:504] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 44387 } } peers { permanent_uuid: "f62c41c9f7ca4279a430e6387349294e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 45729 } } peers { permanent_uuid: "bb44718abfae4e629b47a910e819b6d6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35585 } }
I20260807 08:46:44.336985 15801 leader_election.cc:288] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [CANDIDATE]: Term 1 election: Requested vote from peers f6bb0162daf64b29bc3fbd6ad120ac01 (127.28.108.131:44387), f62c41c9f7ca4279a430e6387349294e (127.28.108.130:45729)
I20260807 08:46:44.337183 15723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e22bbecef14afd83b9bd4aadcc2e17" candidate_uuid: "bb44718abfae4e629b47a910e819b6d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01"
I20260807 08:46:44.337203 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e22bbecef14afd83b9bd4aadcc2e17" candidate_uuid: "bb44718abfae4e629b47a910e819b6d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62c41c9f7ca4279a430e6387349294e"
I20260807 08:46:44.337275 15723 raft_consensus.cc:3037] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:44.337342 15570 raft_consensus.cc:3037] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:44.338214 15723 raft_consensus.cc:2454] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb44718abfae4e629b47a910e819b6d6 in term 1.
I20260807 08:46:44.338428 15344 leader_election.cc:302] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb44718abfae4e629b47a910e819b6d6, f6bb0162daf64b29bc3fbd6ad120ac01; no voters:
I20260807 08:46:44.338474 15570 raft_consensus.cc:2454] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb44718abfae4e629b47a910e819b6d6 in term 1.
I20260807 08:46:44.338550 15801 raft_consensus.cc:2781] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:44.338759 15801 raft_consensus.cc:686] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [term 1 LEADER]: Becoming Leader. State: Replica: bb44718abfae4e629b47a910e819b6d6, State: Running, Role: LEADER
I20260807 08:46:44.338891 15801 consensus_queue.cc:237] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 44387 } } peers { permanent_uuid: "f62c41c9f7ca4279a430e6387349294e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 45729 } } peers { permanent_uuid: "bb44718abfae4e629b47a910e819b6d6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35585 } }
I20260807 08:46:44.339874 15253 catalog_manager.cc:5495] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb44718abfae4e629b47a910e819b6d6 (127.28.108.129). New cstate: current_term: 1 leader_uuid: "bb44718abfae4e629b47a910e819b6d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 44387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f62c41c9f7ca4279a430e6387349294e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 45729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb44718abfae4e629b47a910e819b6d6" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 35585 } health_report { overall_health: HEALTHY } } }
I20260807 08:46:44.371910 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.369225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.28.108.129:35585)
0807 08:46:44.369302 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:44.369304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:44.369316 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:46:44.369372 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:44.369373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:44.369401 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:44.369481 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:46:44.369613 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.370223 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.371012 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.371233 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.371305 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.371520 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:46:44.371646 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:46:44.371718 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:44.371752 (+ 34us) server_negotiation.cc:299] Negotiation successful
0807 08:46:44.371782 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 08:46:44.374460 15570 raft_consensus.cc:1261] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e [term 1 FOLLOWER]: Refusing update from remote peer bb44718abfae4e629b47a910e819b6d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:44.374455 15723 raft_consensus.cc:1261] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01 [term 1 FOLLOWER]: Refusing update from remote peer bb44718abfae4e629b47a910e819b6d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:44.374927 15804 consensus_queue.cc:1035] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62c41c9f7ca4279a430e6387349294e" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:44.375101 15801 consensus_queue.cc:1035] T 62e22bbecef14afd83b9bd4aadcc2e17 P bb44718abfae4e629b47a910e819b6d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6bb0162daf64b29bc3fbd6ad120ac01" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:44.378724 15809 mvcc.cc:205] Tried to move back new op lower bound from 7315834488313851904 to 7315834488172720128. Current Snapshot: MvccSnapshot[applied={T|T < 7315834488313851904}]
I20260807 08:46:44.380908 15253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:44.381034 15253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:44.382037 15253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 536859dd19aa41c4888a9898fd6ffc11: Sending DeleteTablet for 3 replicas of tablet 62e22bbecef14afd83b9bd4aadcc2e17
I20260807 08:46:44.382445 15700 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e22bbecef14afd83b9bd4aadcc2e17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:44 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:50024
I20260807 08:46:44.382552 15546 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e22bbecef14afd83b9bd4aadcc2e17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:44 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:58018
I20260807 08:46:44.382547 15393 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e22bbecef14afd83b9bd4aadcc2e17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:44 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:52934
I20260807 08:46:44.382797 15816 tablet_replica.cc:330] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: stopping tablet replica
I20260807 08:46:44.383030 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 15322
I20260807 08:46:44.389827 15816 raft_consensus.cc:2229] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:44.390082 15816 raft_consensus.cc:2258] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:44.390448 15817 tablet_replica.cc:330] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: stopping tablet replica
I20260807 08:46:44.390620 15817 raft_consensus.cc:2229] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:44.390771 15817 raft_consensus.cc:2258] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:44.390812 15816 ts_tablet_manager.cc:1905] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:44.391352 15817 ts_tablet_manager.cc:1905] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:44.392772 15816 ts_tablet_manager.cc:1918] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:44.392889 15816 log.cc:1196] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/62e22bbecef14afd83b9bd4aadcc2e17
I20260807 08:46:44.392921 15817 ts_tablet_manager.cc:1918] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:44.393020 15817 log.cc:1196] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/62e22bbecef14afd83b9bd4aadcc2e17
I20260807 08:46:44.393323 15817 ts_tablet_manager.cc:1939] T 62e22bbecef14afd83b9bd4aadcc2e17 P f62c41c9f7ca4279a430e6387349294e: Deleting consensus metadata
I20260807 08:46:44.393342 15816 ts_tablet_manager.cc:1939] T 62e22bbecef14afd83b9bd4aadcc2e17 P f6bb0162daf64b29bc3fbd6ad120ac01: Deleting consensus metadata
I20260807 08:46:44.393924 15237 catalog_manager.cc:4841] TS f62c41c9f7ca4279a430e6387349294e (127.28.108.130:45729): tablet 62e22bbecef14afd83b9bd4aadcc2e17 (table test-table [id=83b6b0b480f249d296e747ca2bf27e67]) successfully deleted
I20260807 08:46:44.393986 15239 catalog_manager.cc:4841] TS f6bb0162daf64b29bc3fbd6ad120ac01 (127.28.108.131:44387): tablet 62e22bbecef14afd83b9bd4aadcc2e17 (table test-table [id=83b6b0b480f249d296e747ca2bf27e67]) successfully deleted
W20260807 08:46:44.397810 15237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:35585 (error 108)
I20260807 08:46:44.398504 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 15476
I20260807 08:46:44.398535 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.398383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:35585 (local address 127.0.0.1:52960)
0807 08:46:44.398447 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:44.398490 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:35585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 08:46:44.398697 15237 catalog_manager.cc:4584] TS bb44718abfae4e629b47a910e819b6d6 (127.28.108.129:35585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62e22bbecef14afd83b9bd4aadcc2e17: Network error: Client connection negotiation failed: client connection to 127.28.108.129:35585: connect: Connection refused (error 111)
I20260807 08:46:44.408912 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 15628
I20260807 08:46:44.417960 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 15219
2026-08-07T08:46:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2552 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T08:46:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:46:44Z Disabled control of system clock
I20260807 08:46:44.453501 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:39095
--webserver_interface=127.28.108.190
--webserver_port=0
--builtin_ntp_servers=127.28.108.148:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:39095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 08:46:44.545189 15824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:44.545456 15824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:44.545480 15824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:44.546983 15824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:44.547061 15824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:44.547075 15824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:44.547089 15824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:44.547101 15824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:44.548717 15824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:39095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:39095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:44.549058 15824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:44.549435 15824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:44.555171 15830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:44.555202 15829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:44.555171 15834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:44.555461 15824 server_base.cc:1034] running on GCE node
I20260807 08:46:44.555823 15824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:44.556157 15824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:44.557389 15824 hybrid_clock.cc:648] HybridClock initialized: now 1786092404557358 us; error 56 us; skew 500 ppm
I20260807 08:46:44.564271 15824 webserver.cc:460] Webserver started at http://127.28.108.190:34351/ using document root <none> and password file <none>
I20260807 08:46:44.564733 15824 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:44.564795 15824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:44.564956 15824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:44.566295 15824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "0ba88041c1e443c6a0f2709ae52b0328"
format_stamp: "Formatted at 2026-08-07 08:46:44 on dist-test-slave-9zdj"
server_key: "67e48df99e624b81ce6633a5c0365446"
server_key_iv: "8878d12351ac9a51bbc161213b5b5459"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.566748 15824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "0ba88041c1e443c6a0f2709ae52b0328"
format_stamp: "Formatted at 2026-08-07 08:46:44 on dist-test-slave-9zdj"
server_key: "67e48df99e624b81ce6633a5c0365446"
server_key_iv: "8878d12351ac9a51bbc161213b5b5459"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.568486 15824 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 08:46:44.569841 15840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:44.570266 15824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:44.570379 15824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "0ba88041c1e443c6a0f2709ae52b0328"
format_stamp: "Formatted at 2026-08-07 08:46:44 on dist-test-slave-9zdj"
server_key: "67e48df99e624b81ce6633a5c0365446"
server_key_iv: "8878d12351ac9a51bbc161213b5b5459"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.570657 15824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:44.585546 15824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:44.586024 15824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:44.586217 15824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:44.591239 15824 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:39095
I20260807 08:46:44.591331 15903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:39095 every 8 connection(s)
I20260807 08:46:44.591715 15824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:44.593055 15904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:44.596078 15904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328: Bootstrap starting.
I20260807 08:46:44.596747 15904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:44.597041 15904 log.cc:824] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:44.597939 15904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328: No bootstrap required, opened a new log
I20260807 08:46:44.599977 15904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba88041c1e443c6a0f2709ae52b0328" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39095 } }
I20260807 08:46:44.600163 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 15824
I20260807 08:46:44.600225 15904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:44.600278 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:46:44.600276 15904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba88041c1e443c6a0f2709ae52b0328, State: Initialized, Role: FOLLOWER
I20260807 08:46:44.600492 29106 external_mini_cluster.cc:1428] Setting key 4dcea7d3b44861abe44c198fea1c7e6c
I20260807 08:46:44.600467 15904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba88041c1e443c6a0f2709ae52b0328" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39095 } }
I20260807 08:46:44.600557 15904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:44.600608 15904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:44.600670 15904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:44.601581 15904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba88041c1e443c6a0f2709ae52b0328" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39095 } }
I20260807 08:46:44.601751 15904 leader_election.cc:302] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ba88041c1e443c6a0f2709ae52b0328; no voters:
I20260807 08:46:44.601934 15904 leader_election.cc:288] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:46:44.602006 15909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:44.602219 15909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [term 1 LEADER]: Becoming Leader. State: Replica: 0ba88041c1e443c6a0f2709ae52b0328, State: Running, Role: LEADER
I20260807 08:46:44.602399 15904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:44.602437 15909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba88041c1e443c6a0f2709ae52b0328" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39095 } }
I20260807 08:46:44.602908 15909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ba88041c1e443c6a0f2709ae52b0328. Latest consensus state: current_term: 1 leader_uuid: "0ba88041c1e443c6a0f2709ae52b0328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba88041c1e443c6a0f2709ae52b0328" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39095 } } }
I20260807 08:46:44.603192 15909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:44.603102 15911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ba88041c1e443c6a0f2709ae52b0328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba88041c1e443c6a0f2709ae52b0328" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39095 } } }
I20260807 08:46:44.603729 15911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:44.603942 15920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:44.604823 15920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:44.607360 15920 catalog_manager.cc:1347] Generated new cluster ID: ebcd415ed8f94135bb0fabfb5722f953
I20260807 08:46:44.607443 15920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:46:44.608397 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.601069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.28.108.190:39095)
0807 08:46:44.601381 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:44.601392 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:44.601433 (+ 41us) server_negotiation.cc:407] Connection header received
0807 08:46:44.601577 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:44.601585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:44.601750 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:44.601925 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:44.602231 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.604385 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.605737 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.606363 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.606524 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.607416 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:44.607420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:44.607420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:44.607670 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:44.607718 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:44.607735 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:44.607825 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:44.607986 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 08:46:44.608113 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":137,"threads_started":1}
I20260807 08:46:44.627662 15920 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:46:44.628564 15920 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:44.636174 15920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328: Generated new TSK 0
I20260807 08:46:44.636467 15920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:46:44.639168 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39095
--builtin_ntp_servers=127.28.108.148:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 08:46:44.743597 15932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:44.743916 15932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:44.743969 15932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:44.745958 15932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:44.746033 15932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:44.746068 15932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:46:44.747977 15932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:44.748466 15932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:44.748937 15932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:44.755424 15937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:44.755391 15938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:44.755393 15941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:44.755448 15932 server_base.cc:1034] running on GCE node
I20260807 08:46:44.755927 15932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:44.756238 15932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:44.757464 15932 hybrid_clock.cc:648] HybridClock initialized: now 1786092404757443 us; error 31 us; skew 500 ppm
I20260807 08:46:44.759395 15932 webserver.cc:460] Webserver started at http://127.28.108.129:36473/ using document root <none> and password file <none>
I20260807 08:46:44.759740 15932 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:44.759826 15932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:44.760003 15932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:44.761234 15932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "a3af1da2e56f4d3485991c8de19855f8"
format_stamp: "Formatted at 2026-08-07 08:46:44 on dist-test-slave-9zdj"
server_key: "c2260f09d2fb8a281c1b5b1d3e72544f"
server_key_iv: "da2b0493612d1a82580730e8ee32c367"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.761724 15932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "a3af1da2e56f4d3485991c8de19855f8"
format_stamp: "Formatted at 2026-08-07 08:46:44 on dist-test-slave-9zdj"
server_key: "c2260f09d2fb8a281c1b5b1d3e72544f"
server_key_iv: "da2b0493612d1a82580730e8ee32c367"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.763572 15932 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:46:44.764842 15947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:44.765173 15932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:44.765451 15932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "a3af1da2e56f4d3485991c8de19855f8"
format_stamp: "Formatted at 2026-08-07 08:46:44 on dist-test-slave-9zdj"
server_key: "c2260f09d2fb8a281c1b5b1d3e72544f"
server_key_iv: "da2b0493612d1a82580730e8ee32c367"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.765566 15932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:44.791306 15932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:44.791752 15932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:44.791949 15932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:44.792268 15932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:44.792717 15932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:44.792784 15932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:44.792836 15932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:44.792861 15932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:44.799738 15932 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:41581
I20260807 08:46:44.799769 16083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:41581 every 8 connection(s)
I20260807 08:46:44.800206 15932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:46:44.807070 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.801047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:36407 (local address 127.28.108.190:39095)
0807 08:46:44.801145 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:44.801149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:44.801382 (+ 233us) server_negotiation.cc:407] Connection header received
0807 08:46:44.802113 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:44.802114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:44.802130 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:44.802192 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:44.803031 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.803711 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.805047 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.805313 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.805383 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.806657 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:44.806658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:44.806658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:44.806664 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:44.806680 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:44.806685 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:44.806738 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:44.806964 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 08:46:44.807001 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 08:46:44.807232 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.801012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39095 (local address 127.28.108.129:36407)
0807 08:46:44.801239 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:44.801282 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:44.801932 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:44.802251 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:44.802255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:44.802265 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:44.802927 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.802933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.803915 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.803916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.804930 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.804934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.805432 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.805434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.805584 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.806470 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:46:44.806471 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:44.806507 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:46:44.806535 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:44.806788 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:44.806794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:44.806797 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:46:44.806860 (+ 63us) client_negotiation.cc:240] Negotiation successful
0807 08:46:44.806969 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":121,"threads_started":1}
I20260807 08:46:44.807380 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 15932
I20260807 08:46:44.807482 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:46:44.807766 29106 external_mini_cluster.cc:1428] Setting key e80c2523f8d1a0023631713714587e65
I20260807 08:46:44.808041 16084 heartbeater.cc:347] Connected to a master server at 127.28.108.190:39095
I20260807 08:46:44.808169 16084 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:44.808406 16084 heartbeater.cc:511] Master 127.28.108.190:39095 requested a full tablet report, sending...
I20260807 08:46:44.809031 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39095
--builtin_ntp_servers=127.28.108.148:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 08:46:44.809201 15858 ts_manager.cc:194] Registered new tserver with Master: a3af1da2e56f4d3485991c8de19855f8 (127.28.108.129:41581)
I20260807 08:46:44.810057 15858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.129:36407
W20260807 08:46:44.903264 16087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:44.903604 16087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:44.903649 16087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:44.905210 16087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:44.905304 16087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:44.905354 16087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:46:44.907119 16087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:44.907482 16087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:44.907884 16087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:44.914527 16093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:44.914523 16094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:44.914515 16096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:44.914891 16087 server_base.cc:1034] running on GCE node
I20260807 08:46:44.915136 16087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:44.915503 16087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:44.916764 16087 hybrid_clock.cc:648] HybridClock initialized: now 1786092404916741 us; error 45 us; skew 500 ppm
I20260807 08:46:44.918442 16087 webserver.cc:460] Webserver started at http://127.28.108.130:45509/ using document root <none> and password file <none>
I20260807 08:46:44.918800 16087 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:44.918885 16087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:44.919054 16087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:44.920257 16087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "70d10d14ec6c491891b4eccdef5ac12d"
format_stamp: "Formatted at 2026-08-07 08:46:44 on dist-test-slave-9zdj"
server_key: "78cce05447471cdab87983e0d8a23941"
server_key_iv: "a4962a8c1239ffce73accc298cbe3107"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.920658 16087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "70d10d14ec6c491891b4eccdef5ac12d"
format_stamp: "Formatted at 2026-08-07 08:46:44 on dist-test-slave-9zdj"
server_key: "78cce05447471cdab87983e0d8a23941"
server_key_iv: "a4962a8c1239ffce73accc298cbe3107"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.922238 16087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260807 08:46:44.923272 16103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:44.923555 16087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:44.923684 16087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "70d10d14ec6c491891b4eccdef5ac12d"
format_stamp: "Formatted at 2026-08-07 08:46:44 on dist-test-slave-9zdj"
server_key: "78cce05447471cdab87983e0d8a23941"
server_key_iv: "a4962a8c1239ffce73accc298cbe3107"
server_key_version: "encryptionkey@0"
I20260807 08:46:44.923930 16087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:44.947189 16087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:44.947603 16087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:44.947849 16087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:44.948119 16087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:44.948522 16087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:44.948586 16087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:44.948628 16087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:44.948685 16087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:44.955837 16087 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:40235
I20260807 08:46:44.955940 16227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:40235 every 8 connection(s)
I20260807 08:46:44.956346 16087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:46:44.962822 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.957190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:54817 (local address 127.28.108.190:39095)
0807 08:46:44.957262 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:44.957265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:44.957565 (+ 300us) server_negotiation.cc:407] Connection header received
0807 08:46:44.958228 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:44.958229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:44.958240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:44.958310 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:44.959105 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.959720 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.961068 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.961276 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.961397 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.962491 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:44.962492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:44.962492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:44.962498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:44.962515 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:44.962519 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:44.962568 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:44.962718 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 08:46:44.962748 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 08:46:44.963135 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:44.957171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39095 (local address 127.28.108.130:54817)
0807 08:46:44.957396 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:44.957436 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:44.958090 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:44.958410 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:44.958414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:44.958423 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:44.958999 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.959003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.959845 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.959846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.960955 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:44.960957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:44.961437 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:44.961438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:44.961596 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:44.962325 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:46:44.962325 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:44.962361 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:46:44.962388 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:44.962611 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:44.962620 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:44.962624 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:46:44.962714 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 08:46:44.962824 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":103,"threads_started":1}
I20260807 08:46:44.963824 16228 heartbeater.cc:347] Connected to a master server at 127.28.108.190:39095
I20260807 08:46:44.963960 16228 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:44.964206 16228 heartbeater.cc:511] Master 127.28.108.190:39095 requested a full tablet report, sending...
I20260807 08:46:44.964730 15858 ts_manager.cc:194] Registered new tserver with Master: 70d10d14ec6c491891b4eccdef5ac12d (127.28.108.130:40235)
I20260807 08:46:44.965317 15858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.130:54817
I20260807 08:46:44.966586 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 16087
I20260807 08:46:44.966738 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:46:44.966998 29106 external_mini_cluster.cc:1428] Setting key 52e6ca7e6d6d36f09253a9caf288136b
I20260807 08:46:44.968146 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39095
--builtin_ntp_servers=127.28.108.148:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 08:46:45.067759 16231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:45.068006 16231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:45.068032 16231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:45.069506 16231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:45.069573 16231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:45.069603 16231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:46:45.071240 16231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:39095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:45.071502 16231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:45.071805 16231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:45.077970 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:45.077955 16236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:45.077958 16237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:45.078043 16231 server_base.cc:1034] running on GCE node
I20260807 08:46:45.078438 16231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:45.078845 16231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:45.080057 16231 hybrid_clock.cc:648] HybridClock initialized: now 1786092405080030 us; error 41 us; skew 500 ppm
I20260807 08:46:45.081918 16231 webserver.cc:460] Webserver started at http://127.28.108.131:40231/ using document root <none> and password file <none>
I20260807 08:46:45.082234 16231 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:45.082315 16231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:45.082580 16231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:45.083786 16231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "6b89e683e6a44cc3a9449d97ac624383"
format_stamp: "Formatted at 2026-08-07 08:46:45 on dist-test-slave-9zdj"
server_key: "a2c82c2b122177de9b6628739d05a12e"
server_key_iv: "de0894a723bb90c8f916523fc71edf68"
server_key_version: "encryptionkey@0"
I20260807 08:46:45.084206 16231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "6b89e683e6a44cc3a9449d97ac624383"
format_stamp: "Formatted at 2026-08-07 08:46:45 on dist-test-slave-9zdj"
server_key: "a2c82c2b122177de9b6628739d05a12e"
server_key_iv: "de0894a723bb90c8f916523fc71edf68"
server_key_version: "encryptionkey@0"
I20260807 08:46:45.085948 16231 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 08:46:45.086974 16247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:45.087407 16231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:45.087566 16231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "6b89e683e6a44cc3a9449d97ac624383"
format_stamp: "Formatted at 2026-08-07 08:46:45 on dist-test-slave-9zdj"
server_key: "a2c82c2b122177de9b6628739d05a12e"
server_key_iv: "de0894a723bb90c8f916523fc71edf68"
server_key_version: "encryptionkey@0"
I20260807 08:46:45.087725 16231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:45.108282 16231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:45.108953 16231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:45.109153 16231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:45.109561 16231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:45.110044 16231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:45.110111 16231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:45.110158 16231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:45.110193 16231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:45.117370 16231 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:34859
I20260807 08:46:45.117580 16373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:34859 every 8 connection(s)
I20260807 08:46:45.118024 16231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:46:45.125942 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.119378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:35405 (local address 127.28.108.190:39095)
0807 08:46:45.119480 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:45.119483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:45.119788 (+ 305us) server_negotiation.cc:407] Connection header received
0807 08:46:45.120344 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:45.120346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:45.120360 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:45.120455 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:45.121545 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.122389 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.123596 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.123835 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.123915 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.125486 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:45.125487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:45.125487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:45.125494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:45.125513 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:45.125518 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:45.125576 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:45.125813 (+ 237us) server_negotiation.cc:299] Negotiation successful
0807 08:46:45.125863 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 08:46:45.126170 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 16231
I20260807 08:46:45.126288 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:46:45.126541 29106 external_mini_cluster.cc:1428] Setting key 88e20601380b5df4b14c0259b72f8b04
I20260807 08:46:45.126531 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.119226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:39095 (local address 127.28.108.131:35405)
0807 08:46:45.119574 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:45.119620 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:45.120148 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:45.120576 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:45.120582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:45.120593 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:45.121236 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.121242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.122512 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.122513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.123478 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.123481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.124112 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.124114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.124383 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.125181 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:46:45.125182 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:45.125324 (+ 142us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:46:45.125357 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:45.125592 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:45.125597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:45.125600 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:46:45.125782 (+ 182us) client_negotiation.cc:240] Negotiation successful
0807 08:46:45.125951 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":239,"threads_started":1}
I20260807 08:46:45.127239 16374 heartbeater.cc:347] Connected to a master server at 127.28.108.190:39095
I20260807 08:46:45.127408 16374 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:45.127699 16374 heartbeater.cc:511] Master 127.28.108.190:39095 requested a full tablet report, sending...
I20260807 08:46:45.128161 15858 ts_manager.cc:194] Registered new tserver with Master: 6b89e683e6a44cc3a9449d97ac624383 (127.28.108.131:34859)
I20260807 08:46:45.128695 15858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.131:35405
I20260807 08:46:45.139277 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:46:45.144599 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.140978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.28.108.190:39095)
0807 08:46:45.141050 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:45.141052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:45.141176 (+ 124us) server_negotiation.cc:407] Connection header received
0807 08:46:45.141210 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:45.141211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:45.141237 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:45.141338 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:45.141821 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.142390 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.143143 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.143364 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.143428 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.144312 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:45.144314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:45.144314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:45.144319 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:45.144334 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:45.144339 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:45.144399 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:45.144501 (+ 102us) server_negotiation.cc:299] Negotiation successful
0807 08:46:45.144530 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 08:46:45.146775 15858 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:46:45.148550 15858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:46:45.158370 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.152047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.28.108.129:41581)
0807 08:46:45.152503 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:45.152510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:45.152806 (+ 296us) server_negotiation.cc:407] Connection header received
0807 08:46:45.152863 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:45.152866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:45.152885 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:45.152968 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:45.153277 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.154752 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.156638 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.157471 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.157519 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.157648 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:45.158032 (+ 384us) server_negotiation.cc:299] Negotiation successful
0807 08:46:45.158084 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":297,"threads_started":1}
I20260807 08:46:45.158696 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.151923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:41581 (local address 127.0.0.1:34694)
0807 08:46:45.152520 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:45.152578 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:45.152693 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:45.153068 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:45.153071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:45.153074 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:45.153175 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.153178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.154872 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.154873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.156523 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.156526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.157765 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.157766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.157886 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.157902 (+ 16us) client_negotiation.cc:769] Sending connection context
0807 08:46:45.158512 (+ 610us) client_negotiation.cc:240] Negotiation successful
0807 08:46:45.158611 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":78,"threads_started":1}
I20260807 08:46:45.159650 16005 tablet_service.cc:1467] Processing CreateTablet for tablet f0f9a13c67ae4fddb54719b4227aeba0 (DEFAULT_TABLE table=test-table [id=fe56bca082c9409f90b69889dae3e7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:45.160164 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f9a13c67ae4fddb54719b4227aeba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:45.162060 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.152193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:40235 (local address 127.0.0.1:57706)
0807 08:46:45.152918 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:45.152930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:45.152995 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:45.155732 (+ 2737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:45.155734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:45.155736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:45.155787 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.155789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.159181 (+ 3392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.159183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.160450 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.160455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.161696 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.161698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.161766 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.161772 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:46:45.161846 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 08:46:45.161926 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":98,"threads_started":1}
I20260807 08:46:45.162317 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.152396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:34859 (local address 127.0.0.1:33696)
0807 08:46:45.153266 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:45.153278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:45.153336 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:45.155263 (+ 1927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:45.155264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:45.155267 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:45.155327 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.155329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.157388 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.157389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.159739 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.159743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.161200 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.161201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.161276 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.161282 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:46:45.161822 (+ 540us) client_negotiation.cc:240] Negotiation successful
0807 08:46:45.161900 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":77,"threads_started":1}
I20260807 08:46:45.162667 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.152667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.28.108.130:40235)
0807 08:46:45.155448 (+ 2781us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:45.155453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:45.155465 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:46:45.155512 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:45.155514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:45.155531 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:45.155642 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:45.155936 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.158780 (+ 2844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.160574 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.161459 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.161534 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.161654 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:45.162492 (+ 838us) server_negotiation.cc:299] Negotiation successful
0807 08:46:45.162561 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2727,"thread_start_us":180,"threads_started":1}
I20260807 08:46:45.162667 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.152720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.28.108.131:34859)
0807 08:46:45.154988 (+ 2268us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:45.154995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:45.155009 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:46:45.155060 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:45.155063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:45.155084 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:45.155280 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:45.156003 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.157146 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.159866 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.160962 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.161053 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.161155 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:45.161986 (+ 831us) server_negotiation.cc:299] Negotiation successful
0807 08:46:45.162051 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2202,"thread_start_us":98,"threads_started":1}
I20260807 08:46:45.163452 16154 tablet_service.cc:1467] Processing CreateTablet for tablet f0f9a13c67ae4fddb54719b4227aeba0 (DEFAULT_TABLE table=test-table [id=fe56bca082c9409f90b69889dae3e7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:45.163465 16301 tablet_service.cc:1467] Processing CreateTablet for tablet f0f9a13c67ae4fddb54719b4227aeba0 (DEFAULT_TABLE table=test-table [id=fe56bca082c9409f90b69889dae3e7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:45.163854 16301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f9a13c67ae4fddb54719b4227aeba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:45.163849 16154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f9a13c67ae4fddb54719b4227aeba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:45.164662 16394 tablet_bootstrap.cc:492] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8: Bootstrap starting.
I20260807 08:46:45.165829 16394 tablet_bootstrap.cc:654] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:45.166253 16394 log.cc:824] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:45.167748 16397 tablet_bootstrap.cc:492] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: Bootstrap starting.
I20260807 08:46:45.168542 16396 tablet_bootstrap.cc:492] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383: Bootstrap starting.
I20260807 08:46:45.168747 16394 tablet_bootstrap.cc:492] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8: No bootstrap required, opened a new log
I20260807 08:46:45.168826 16394 ts_tablet_manager.cc:1397] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260807 08:46:45.168835 16397 tablet_bootstrap.cc:654] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:45.169332 16397 log.cc:824] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:45.169543 16396 tablet_bootstrap.cc:654] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:45.170090 16396 log.cc:824] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:45.170997 16397 tablet_bootstrap.cc:492] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: No bootstrap required, opened a new log
I20260807 08:46:45.171146 16397 ts_tablet_manager.cc:1397] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 08:46:45.171299 16396 tablet_bootstrap.cc:492] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383: No bootstrap required, opened a new log
I20260807 08:46:45.171394 16396 ts_tablet_manager.cc:1397] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:46:45.172098 16394 raft_consensus.cc:348] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af1da2e56f4d3485991c8de19855f8" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41581 } } peers { permanent_uuid: "70d10d14ec6c491891b4eccdef5ac12d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 40235 } } peers { permanent_uuid: "6b89e683e6a44cc3a9449d97ac624383" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34859 } }
I20260807 08:46:45.172340 16394 raft_consensus.cc:374] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:45.172389 16394 raft_consensus.cc:729] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3af1da2e56f4d3485991c8de19855f8, State: Initialized, Role: FOLLOWER
I20260807 08:46:45.172544 16394 consensus_queue.cc:260] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af1da2e56f4d3485991c8de19855f8" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41581 } } peers { permanent_uuid: "70d10d14ec6c491891b4eccdef5ac12d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 40235 } } peers { permanent_uuid: "6b89e683e6a44cc3a9449d97ac624383" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34859 } }
I20260807 08:46:45.172873 16394 ts_tablet_manager.cc:1428] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260807 08:46:45.173025 16084 heartbeater.cc:503] Master 127.28.108.190:39095 was elected leader, sending a full tablet report...
I20260807 08:46:45.174768 16397 raft_consensus.cc:348] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af1da2e56f4d3485991c8de19855f8" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41581 } } peers { permanent_uuid: "70d10d14ec6c491891b4eccdef5ac12d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 40235 } } peers { permanent_uuid: "6b89e683e6a44cc3a9449d97ac624383" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34859 } }
I20260807 08:46:45.175001 16397 raft_consensus.cc:374] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:45.175051 16397 raft_consensus.cc:729] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d10d14ec6c491891b4eccdef5ac12d, State: Initialized, Role: FOLLOWER
I20260807 08:46:45.175191 16397 consensus_queue.cc:260] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af1da2e56f4d3485991c8de19855f8" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41581 } } peers { permanent_uuid: "70d10d14ec6c491891b4eccdef5ac12d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 40235 } } peers { permanent_uuid: "6b89e683e6a44cc3a9449d97ac624383" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34859 } }
I20260807 08:46:45.175520 16397 ts_tablet_manager.cc:1428] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 08:46:45.175653 16228 heartbeater.cc:503] Master 127.28.108.190:39095 was elected leader, sending a full tablet report...
I20260807 08:46:45.175143 16396 raft_consensus.cc:348] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af1da2e56f4d3485991c8de19855f8" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41581 } } peers { permanent_uuid: "70d10d14ec6c491891b4eccdef5ac12d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 40235 } } peers { permanent_uuid: "6b89e683e6a44cc3a9449d97ac624383" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34859 } }
I20260807 08:46:45.176050 16396 raft_consensus.cc:374] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:45.176107 16396 raft_consensus.cc:729] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b89e683e6a44cc3a9449d97ac624383, State: Initialized, Role: FOLLOWER
I20260807 08:46:45.176259 16396 consensus_queue.cc:260] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af1da2e56f4d3485991c8de19855f8" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41581 } } peers { permanent_uuid: "70d10d14ec6c491891b4eccdef5ac12d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 40235 } } peers { permanent_uuid: "6b89e683e6a44cc3a9449d97ac624383" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34859 } }
I20260807 08:46:45.176541 16396 ts_tablet_manager.cc:1428] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 08:46:45.176577 16374 heartbeater.cc:503] Master 127.28.108.190:39095 was elected leader, sending a full tablet report...
W20260807 08:46:45.207077 16229 tablet.cc:2369] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:46:45.301081 16085 tablet.cc:2369] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:46:45.304275 16400 raft_consensus.cc:482] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:45.304472 16400 raft_consensus.cc:504] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af1da2e56f4d3485991c8de19855f8" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41581 } } peers { permanent_uuid: "70d10d14ec6c491891b4eccdef5ac12d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 40235 } } peers { permanent_uuid: "6b89e683e6a44cc3a9449d97ac624383" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34859 } }
I20260807 08:46:45.304879 16400 leader_election.cc:288] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70d10d14ec6c491891b4eccdef5ac12d (127.28.108.130:40235), 6b89e683e6a44cc3a9449d97ac624383 (127.28.108.131:34859)
I20260807 08:46:45.308585 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.305529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:34859 (local address 127.28.108.129:47871)
0807 08:46:45.305691 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:45.305777 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:45.305855 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:45.306018 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:45.306019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:45.306020 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:45.306107 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.306108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.306790 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.306791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.307658 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.307661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.308271 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.308272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.308367 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.308371 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:46:45.308443 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 08:46:45.308489 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":29,"thread_start_us":113,"threads_started":1}
I20260807 08:46:45.308571 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.305607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:47871 (local address 127.28.108.131:34859)
0807 08:46:45.305695 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:45.305697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:45.305850 (+ 153us) server_negotiation.cc:407] Connection header received
0807 08:46:45.305916 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:45.305917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:45.305928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:45.306000 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:45.306181 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.306697 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.307770 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.308152 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.308267 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.308277 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:45.308453 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 08:46:45.308498 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 08:46:45.309052 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.305308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:40235 (local address 127.28.108.129:56357)
0807 08:46:45.305405 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:45.305420 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:45.305510 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:45.305765 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:45.305766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:45.305768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:45.305845 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.305847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.306778 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.306780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.307791 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:45.307795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.308792 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.308793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:45.308860 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.308863 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:46:45.308938 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 08:46:45.308984 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":50}
I20260807 08:46:45.309060 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.305519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:56357 (local address 127.28.108.130:40235)
0807 08:46:45.305579 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:45.305581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:45.305592 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:46:45.305649 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:45.305650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:45.305662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:45.305732 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:45.305928 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.306434 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.308166 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.308669 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.308809 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.308821 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:45.308948 (+ 127us) server_negotiation.cc:299] Negotiation successful
0807 08:46:45.308998 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 08:46:45.309248 16321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f9a13c67ae4fddb54719b4227aeba0" candidate_uuid: "a3af1da2e56f4d3485991c8de19855f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b89e683e6a44cc3a9449d97ac624383" is_pre_election: true
I20260807 08:46:45.309477 16321 raft_consensus.cc:2454] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3af1da2e56f4d3485991c8de19855f8 in term 0.
I20260807 08:46:45.309857 15948 leader_election.cc:302] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b89e683e6a44cc3a9449d97ac624383, a3af1da2e56f4d3485991c8de19855f8; no voters:
I20260807 08:46:45.310001 16400 raft_consensus.cc:2781] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:46:45.310068 16400 raft_consensus.cc:482] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:46:45.310009 16178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f9a13c67ae4fddb54719b4227aeba0" candidate_uuid: "a3af1da2e56f4d3485991c8de19855f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d10d14ec6c491891b4eccdef5ac12d" is_pre_election: true
I20260807 08:46:45.310086 16400 raft_consensus.cc:3037] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:45.310184 16178 raft_consensus.cc:2454] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3af1da2e56f4d3485991c8de19855f8 in term 0.
I20260807 08:46:45.311051 16400 raft_consensus.cc:504] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af1da2e56f4d3485991c8de19855f8" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41581 } } peers { permanent_uuid: "70d10d14ec6c491891b4eccdef5ac12d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 40235 } } peers { permanent_uuid: "6b89e683e6a44cc3a9449d97ac624383" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34859 } }
I20260807 08:46:45.311218 16400 leader_election.cc:288] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [CANDIDATE]: Term 1 election: Requested vote from peers 70d10d14ec6c491891b4eccdef5ac12d (127.28.108.130:40235), 6b89e683e6a44cc3a9449d97ac624383 (127.28.108.131:34859)
I20260807 08:46:45.311414 16321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f9a13c67ae4fddb54719b4227aeba0" candidate_uuid: "a3af1da2e56f4d3485991c8de19855f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b89e683e6a44cc3a9449d97ac624383"
I20260807 08:46:45.311480 16178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f9a13c67ae4fddb54719b4227aeba0" candidate_uuid: "a3af1da2e56f4d3485991c8de19855f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d10d14ec6c491891b4eccdef5ac12d"
I20260807 08:46:45.311517 16321 raft_consensus.cc:3037] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:45.311589 16178 raft_consensus.cc:3037] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:45.312352 16178 raft_consensus.cc:2454] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3af1da2e56f4d3485991c8de19855f8 in term 1.
I20260807 08:46:45.312376 16321 raft_consensus.cc:2454] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3af1da2e56f4d3485991c8de19855f8 in term 1.
I20260807 08:46:45.312588 15948 leader_election.cc:302] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b89e683e6a44cc3a9449d97ac624383, a3af1da2e56f4d3485991c8de19855f8; no voters:
I20260807 08:46:45.312685 16400 raft_consensus.cc:2781] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:45.312860 16400 raft_consensus.cc:686] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 1 LEADER]: Becoming Leader. State: Replica: a3af1da2e56f4d3485991c8de19855f8, State: Running, Role: LEADER
I20260807 08:46:45.312988 16400 consensus_queue.cc:237] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af1da2e56f4d3485991c8de19855f8" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41581 } } peers { permanent_uuid: "70d10d14ec6c491891b4eccdef5ac12d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 40235 } } peers { permanent_uuid: "6b89e683e6a44cc3a9449d97ac624383" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34859 } }
I20260807 08:46:45.314043 15857 catalog_manager.cc:5495] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3af1da2e56f4d3485991c8de19855f8 (127.28.108.129). New cstate: current_term: 1 leader_uuid: "a3af1da2e56f4d3485991c8de19855f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af1da2e56f4d3485991c8de19855f8" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70d10d14ec6c491891b4eccdef5ac12d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 40235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b89e683e6a44cc3a9449d97ac624383" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260807 08:46:45.345793 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.342822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.28.108.129:41581)
0807 08:46:45.342901 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:45.342903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:45.342912 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:46:45.342965 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:45.342966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:45.343012 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:45.343094 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:46:45.343248 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.343802 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.344668 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:45.344912 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:45.344996 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:45.345231 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:46:45.345456 (+ 225us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:46:45.345579 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:45.345600 (+ 21us) server_negotiation.cc:299] Negotiation successful
0807 08:46:45.345625 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 08:46:45.347998 16178 raft_consensus.cc:1261] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d [term 1 FOLLOWER]: Refusing update from remote peer a3af1da2e56f4d3485991c8de19855f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:45.348028 16321 raft_consensus.cc:1261] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383 [term 1 FOLLOWER]: Refusing update from remote peer a3af1da2e56f4d3485991c8de19855f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:45.348557 16400 consensus_queue.cc:1035] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b89e683e6a44cc3a9449d97ac624383" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:45.348793 16404 consensus_queue.cc:1035] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70d10d14ec6c491891b4eccdef5ac12d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 40235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:45.352106 16409 mvcc.cc:205] Tried to move back new op lower bound from 7315834492301914112 to 7315834492162740224. Current Snapshot: MvccSnapshot[applied={T|T < 7315834492301914112}]
I20260807 08:46:45.355331 15858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:45.355449 15858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:45.356454 15858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ba88041c1e443c6a0f2709ae52b0328: Sending DeleteTablet for 3 replicas of tablet f0f9a13c67ae4fddb54719b4227aeba0
I20260807 08:46:45.356841 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f9a13c67ae4fddb54719b4227aeba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:45 UTC) from {username='slave'} at 127.0.0.1:34694
I20260807 08:46:45.357028 16154 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f9a13c67ae4fddb54719b4227aeba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:45 UTC) from {username='slave'} at 127.0.0.1:57706
I20260807 08:46:45.357146 16418 tablet_replica.cc:330] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8: stopping tablet replica
I20260807 08:46:45.357326 16301 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f9a13c67ae4fddb54719b4227aeba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:45 UTC) from {username='slave'} at 127.0.0.1:33696
I20260807 08:46:45.357283 16418 raft_consensus.cc:2229] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:46:45.357513 16418 raft_consensus.cc:2258] T f0f9a13c67ae4fddb54719b4227aeba0 P a3af1da2e56f4d3485991c8de19855f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:45.357663 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 15932
I20260807 08:46:45.357749 16419 tablet_replica.cc:330] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: stopping tablet replica
I20260807 08:46:45.358668 16420 tablet_replica.cc:330] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383: stopping tablet replica
I20260807 08:46:45.358758 16420 raft_consensus.cc:2229] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:45.358856 16420 raft_consensus.cc:2258] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:45.359023 16419 raft_consensus.cc:2229] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:45.359129 16419 raft_consensus.cc:2258] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:45.359630 16420 ts_tablet_manager.cc:1905] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:45.359630 16419 ts_tablet_manager.cc:1905] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:45.361783 16420 ts_tablet_manager.cc:1918] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:45.361783 16419 ts_tablet_manager.cc:1918] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:45.361894 16419 log.cc:1196] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/f0f9a13c67ae4fddb54719b4227aeba0
I20260807 08:46:45.361894 16420 log.cc:1196] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/f0f9a13c67ae4fddb54719b4227aeba0
I20260807 08:46:45.362214 16420 ts_tablet_manager.cc:1939] T f0f9a13c67ae4fddb54719b4227aeba0 P 6b89e683e6a44cc3a9449d97ac624383: Deleting consensus metadata
I20260807 08:46:45.362215 16419 ts_tablet_manager.cc:1939] T f0f9a13c67ae4fddb54719b4227aeba0 P 70d10d14ec6c491891b4eccdef5ac12d: Deleting consensus metadata
I20260807 08:46:45.362783 15841 catalog_manager.cc:4841] TS 70d10d14ec6c491891b4eccdef5ac12d (127.28.108.130:40235): tablet f0f9a13c67ae4fddb54719b4227aeba0 (table test-table [id=fe56bca082c9409f90b69889dae3e7d1]) successfully deleted
I20260807 08:46:45.362910 15841 catalog_manager.cc:4841] TS 6b89e683e6a44cc3a9449d97ac624383 (127.28.108.131:34859): tablet f0f9a13c67ae4fddb54719b4227aeba0 (table test-table [id=fe56bca082c9409f90b69889dae3e7d1]) successfully deleted
W20260807 08:46:45.366755 15841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:41581 (error 108)
I20260807 08:46:45.367522 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:45.367336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:41581 (local address 127.0.0.1:34712)
0807 08:46:45.367418 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:45.367486 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 08:46:45.367530 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 16087
W20260807 08:46:45.370687 15841 catalog_manager.cc:4584] TS a3af1da2e56f4d3485991c8de19855f8 (127.28.108.129:41581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0f9a13c67ae4fddb54719b4227aeba0: Network error: Client connection negotiation failed: client connection to 127.28.108.129:41581: connect: Connection refused (error 111)
I20260807 08:46:45.375808 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 16231
I20260807 08:46:45.383066 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 15824
2026-08-07T08:46:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (964 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:46:45 dist-test-slave-9zdj krb5kdc[16424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:46:45 dist-test-slave-9zdj krb5kdc[16424](info): set up 2 sockets
Aug 07 08:46:45 dist-test-slave-9zdj krb5kdc[16424](info): commencing operation
krb5kdc: starting...
W20260807 08:46:46.547504 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:46:46 dist-test-slave-9zdj krb5kdc[16424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:46:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:46:46.705793 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:41917
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:46.796325 16438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:46.796645 16438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:46.796691 16438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:46.798055 16438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:46.798173 16438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:46.798202 16438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:46.798243 16438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:46.798272 16438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:46.800124 16438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:41917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:46.800527 16438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:46.800807 16438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:46.806528 16448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:46.806643 16445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:46.806528 16444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:46.807271 16438 server_base.cc:1034] running on GCE node
I20260807 08:46:46.807596 16438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:46.808007 16438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:46.809284 16438 hybrid_clock.cc:648] HybridClock initialized: now 1786092406809229 us; error 69 us; skew 500 ppm
Aug 07 08:46:46 dist-test-slave-9zdj krb5kdc[16424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092406, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:46.812608 16438 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:46:46.813234 16438 webserver.cc:460] Webserver started at http://127.28.108.190:41281/ using document root <none> and password file <none>
I20260807 08:46:46.813508 16438 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:46.813562 16438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:46.813819 16438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:46.815037 16438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "5ca6a5a1d71048fc953fc0e772382dba"
format_stamp: "Formatted at 2026-08-07 08:46:46 on dist-test-slave-9zdj"
server_key: "4cdb56dd9ffe92cb14270f49272503bc"
server_key_iv: "8f2520f922d91cc6c11735ac42a6d5e3"
server_key_version: "encryptionkey@0"
I20260807 08:46:46.815403 16438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "5ca6a5a1d71048fc953fc0e772382dba"
format_stamp: "Formatted at 2026-08-07 08:46:46 on dist-test-slave-9zdj"
server_key: "4cdb56dd9ffe92cb14270f49272503bc"
server_key_iv: "8f2520f922d91cc6c11735ac42a6d5e3"
server_key_version: "encryptionkey@0"
I20260807 08:46:46.816962 16438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260807 08:46:46.817935 16455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:46.818305 16438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:46.818403 16438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "5ca6a5a1d71048fc953fc0e772382dba"
format_stamp: "Formatted at 2026-08-07 08:46:46 on dist-test-slave-9zdj"
server_key: "4cdb56dd9ffe92cb14270f49272503bc"
server_key_iv: "8f2520f922d91cc6c11735ac42a6d5e3"
server_key_version: "encryptionkey@0"
I20260807 08:46:46.818516 16438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:46.833035 16438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:46.835994 16438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:46.836278 16438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:46.841459 16438 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:41917
I20260807 08:46:46.841508 16514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:41917 every 8 connection(s)
I20260807 08:46:46.841974 16438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:46.843218 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 16438
I20260807 08:46:46.843369 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:46:46.843393 16515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:46.843633 29106 external_mini_cluster.cc:1428] Setting key 66f17cf7b5d4b8e13e0d25630d0f2996
I20260807 08:46:46.847561 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba: Bootstrap starting.
I20260807 08:46:46.848482 16515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:46.848788 16515 log.cc:824] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:46.849807 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba: No bootstrap required, opened a new log
Aug 07 08:46:46 dist-test-slave-9zdj krb5kdc[16424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:46.852790 16515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6a5a1d71048fc953fc0e772382dba" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41917 } }
I20260807 08:46:46.853050 16515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:46.853138 16515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca6a5a1d71048fc953fc0e772382dba, State: Initialized, Role: FOLLOWER
I20260807 08:46:46.853313 16515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6a5a1d71048fc953fc0e772382dba" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41917 } }
I20260807 08:46:46.853385 16515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:46.853411 16515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:46.853488 16515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:46.854576 16515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6a5a1d71048fc953fc0e772382dba" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41917 } }
I20260807 08:46:46.854727 16515 leader_election.cc:302] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ca6a5a1d71048fc953fc0e772382dba; no voters:
I20260807 08:46:46.854941 16515 leader_election.cc:288] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:46:46.854976 16520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:46.855136 16520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [term 1 LEADER]: Becoming Leader. State: Replica: 5ca6a5a1d71048fc953fc0e772382dba, State: Running, Role: LEADER
I20260807 08:46:46.855278 16520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6a5a1d71048fc953fc0e772382dba" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41917 } }
I20260807 08:46:46.855312 16515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:46.855762 16521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ca6a5a1d71048fc953fc0e772382dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6a5a1d71048fc953fc0e772382dba" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41917 } } }
I20260807 08:46:46.855789 16523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ca6a5a1d71048fc953fc0e772382dba. Latest consensus state: current_term: 1 leader_uuid: "5ca6a5a1d71048fc953fc0e772382dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6a5a1d71048fc953fc0e772382dba" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41917 } } }
I20260807 08:46:46.855893 16523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:46.855850 16521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:46.856763 16526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:46.857434 16526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:46.858399 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:46.844379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.28.108.190:41917)
0807 08:46:46.844654 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:46.844667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:46.844709 (+ 42us) server_negotiation.cc:407] Connection header received
0807 08:46:46.845291 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:46.845303 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:46.845380 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:46.845569 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:46.846581 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:46.848184 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:46.849453 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:46.850195 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:46.850340 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:46.853828 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:46.853834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:46.853834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:46.853866 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:46.856313 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:46.856912 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:46.856916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:46.856917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:46.857048 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:46.857311 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:46.857312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:46.857313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:46.857702 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:46.857886 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:46.858024 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 08:46:46.858187 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":113,"threads_started":1}
I20260807 08:46:46.859650 16526 catalog_manager.cc:1347] Generated new cluster ID: b9a555f671e34bcf9e67d2d7b2a2ed95
I20260807 08:46:46.859714 16526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:46:46.878665 16526 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:46:46.879395 16526 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:46.885177 16526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba: Generated new TSK 0
I20260807 08:46:46.885496 16526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:46:46.927126 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41917
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:47.030572 16543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:47.030938 16543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:47.030972 16543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:47.032227 16543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:47.032274 16543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:47.032303 16543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:46:47.033983 16543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:47.034235 16543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:47.034490 16543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:47.040722 16548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:47.040825 16552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:47.040727 16550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:47.041446 16543 server_base.cc:1034] running on GCE node
I20260807 08:46:47.041729 16543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:47.042055 16543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:47.043228 16543 hybrid_clock.cc:648] HybridClock initialized: now 1786092407043210 us; error 30 us; skew 500 ppm
Aug 07 08:46:47 dist-test-slave-9zdj krb5kdc[16424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092407, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:47.046433 16543 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:46:47.046904 16543 webserver.cc:460] Webserver started at http://127.28.108.129:42283/ using document root <none> and password file <none>
I20260807 08:46:47.047107 16543 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:47.047152 16543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:47.047256 16543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:47.048482 16543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "73404b6f29194826bdcbdb7a36418147"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "0b756d5bb41e8be3742a819f29e51b8a"
server_key_iv: "0ea6cb2d55584691731211c43d1a5906"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.048830 16543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "73404b6f29194826bdcbdb7a36418147"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "0b756d5bb41e8be3742a819f29e51b8a"
server_key_iv: "0ea6cb2d55584691731211c43d1a5906"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.050736 16543 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 08:46:47.051844 16559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:47.052088 16543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:47.052204 16543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "73404b6f29194826bdcbdb7a36418147"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "0b756d5bb41e8be3742a819f29e51b8a"
server_key_iv: "0ea6cb2d55584691731211c43d1a5906"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.052312 16543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:47.074376 16543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:47.077113 16543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:47.077283 16543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:47.077790 16543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:47.078166 16543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:47.078207 16543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:47.078356 16543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:47.078374 16543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:47.085901 16543 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:40735
I20260807 08:46:47.086004 16681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:40735 every 8 connection(s)
I20260807 08:46:47.086452 16543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
Aug 07 08:46:47 dist-test-slave-9zdj krb5kdc[16424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092407, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:47.095963 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 16543
I20260807 08:46:47.096133 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:46:47.096453 29106 external_mini_cluster.cc:1428] Setting key 215f47719e34a1c95e00abb503cf31a0
I20260807 08:46:47.099480 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.087685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:46881 (local address 127.28.108.190:41917)
0807 08:46:47.087782 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:47.087784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:47.088051 (+ 267us) server_negotiation.cc:407] Connection header received
0807 08:46:47.088662 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:47.088664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:47.088675 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:47.088739 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:47.089997 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.090550 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.091802 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.092052 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.092131 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.095080 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:47.095082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:47.095082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:47.095091 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:47.097316 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:47.098028 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:47.098029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:47.098030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:47.098106 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:47.098568 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:47.098569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:47.098569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:47.098763 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:47.098833 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:47.099181 (+ 348us) server_negotiation.cc:299] Negotiation successful
0807 08:46:47.099324 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 08:46:47.099622 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.087650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41917 (local address 127.28.108.129:46881)
0807 08:46:47.087886 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:47.087942 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:47.088534 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:47.088844 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:47.088848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:47.089584 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:47.089862 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.089868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.090673 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.090675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.091687 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.091690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.092169 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.092170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.092313 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.093041 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:47.093052 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:47.094950 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:47.097441 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:47.097458 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:47.097461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:47.097936 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:47.098212 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:47.098213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:47.098213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:47.098469 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:47.098835 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:47.098842 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:47.099002 (+ 160us) client_negotiation.cc:769] Sending connection context
0807 08:46:47.099118 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 08:46:47.099360 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":135,"threads_started":1}
I20260807 08:46:47.100216 16682 heartbeater.cc:347] Connected to a master server at 127.28.108.190:41917
I20260807 08:46:47.100378 16682 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:47.100611 16682 heartbeater.cc:511] Master 127.28.108.190:41917 requested a full tablet report, sending...
I20260807 08:46:47.101361 16474 ts_manager.cc:194] Registered new tserver with Master: 73404b6f29194826bdcbdb7a36418147 (127.28.108.129:40735)
I20260807 08:46:47.102383 16474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:46881
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:46:47.134948 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41917
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:47.227109 16688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:47.227602 16688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:47.227662 16688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:47.228966 16688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:47.229069 16688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:47.229123 16688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:46:47.230845 16688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:47.231161 16688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:47.231438 16688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:47.238327 16698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:47.238355 16695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:47.238334 16696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:47.238665 16688 server_base.cc:1034] running on GCE node
I20260807 08:46:47.238906 16688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:47.239240 16688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:47.240442 16688 hybrid_clock.cc:648] HybridClock initialized: now 1786092407240398 us; error 58 us; skew 500 ppm
Aug 07 08:46:47 dist-test-slave-9zdj krb5kdc[16424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092407, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:47.243907 16688 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:46:47.244587 16688 webserver.cc:460] Webserver started at http://127.28.108.130:34277/ using document root <none> and password file <none>
I20260807 08:46:47.244935 16688 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:47.245028 16688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:47.245245 16688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:47.246634 16688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "d502af8ad8224d68a811e6f66f57e432"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "4d958edfca591d11bb7757375d76865b"
server_key_iv: "19b43d61212a96f57548e7bd155757c2"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.247123 16688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "d502af8ad8224d68a811e6f66f57e432"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "4d958edfca591d11bb7757375d76865b"
server_key_iv: "19b43d61212a96f57548e7bd155757c2"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.249066 16688 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 08:46:47.250608 16707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:47.250953 16688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:47.251088 16688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "d502af8ad8224d68a811e6f66f57e432"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "4d958edfca591d11bb7757375d76865b"
server_key_iv: "19b43d61212a96f57548e7bd155757c2"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.251185 16688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:47.269552 16688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:47.272250 16688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:47.272454 16688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:47.272740 16688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:47.273109 16688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:47.273167 16688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:47.273207 16688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:47.273255 16688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:47.280472 16688 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:41127
I20260807 08:46:47.280550 16829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:41127 every 8 connection(s)
I20260807 08:46:47.281025 16688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:46:47.282562 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 16688
I20260807 08:46:47.282672 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:46:47.282923 29106 external_mini_cluster.cc:1428] Setting key 67bfa4f5e073373b915d7d1d775cac71
Aug 07 08:46:47 dist-test-slave-9zdj krb5kdc[16424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092407, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
I20260807 08:46:47.293851 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.282101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:54023 (local address 127.28.108.190:41917)
0807 08:46:47.282164 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:47.282166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:47.282484 (+ 318us) server_negotiation.cc:407] Connection header received
0807 08:46:47.283192 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:47.283194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:47.283208 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:47.283268 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:47.284838 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.285446 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.286733 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.286969 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.287072 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.290090 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:47.290093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:47.290093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:47.290103 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:47.292003 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:47.292622 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:47.292624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:47.292625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:47.292708 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:47.293007 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:47.293008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:47.293008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:47.293205 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:47.293273 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:47.293610 (+ 337us) server_negotiation.cc:299] Negotiation successful
0807 08:46:47.293744 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260807 08:46:47.294157 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.282094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41917 (local address 127.28.108.130:54023)
0807 08:46:47.282305 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:47.282366 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:47.283074 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:47.283376 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:47.283381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:47.284364 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:47.284686 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.284694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.285573 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.285574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.286617 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.286620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.287088 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.287090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.287252 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.287873 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:47.287882 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:47.289937 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:47.292220 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:47.292232 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:47.292234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:47.292515 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:47.292788 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:47.292789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:47.292789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:47.292931 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:47.293263 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:47.293272 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:47.293453 (+ 181us) client_negotiation.cc:769] Sending connection context
0807 08:46:47.293588 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 08:46:47.293820 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":105,"threads_started":1}
I20260807 08:46:47.295105 16830 heartbeater.cc:347] Connected to a master server at 127.28.108.190:41917
I20260807 08:46:47.295279 16830 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:47.295529 16830 heartbeater.cc:511] Master 127.28.108.190:41917 requested a full tablet report, sending...
I20260807 08:46:47.296049 16474 ts_manager.cc:194] Registered new tserver with Master: d502af8ad8224d68a811e6f66f57e432 (127.28.108.130:41127)
I20260807 08:46:47.296518 16474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:54023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:46:47.318488 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41917
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:47.413074 16835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:47.413338 16835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:47.413367 16835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:47.414794 16835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:47.414860 16835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:47.414891 16835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:46:47.416565 16835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:47.416884 16835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:47.417166 16835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:47.423609 16841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:47.423616 16842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:47.423626 16844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:47.424052 16835 server_base.cc:1034] running on GCE node
I20260807 08:46:47.424331 16835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:47.424674 16835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:47.425855 16835 hybrid_clock.cc:648] HybridClock initialized: now 1786092407425827 us; error 48 us; skew 500 ppm
Aug 07 08:46:47 dist-test-slave-9zdj krb5kdc[16424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092407, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:47.428912 16835 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:46:47.429454 16835 webserver.cc:460] Webserver started at http://127.28.108.131:34781/ using document root <none> and password file <none>
I20260807 08:46:47.429790 16835 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:47.429867 16835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:47.430063 16835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:47.431231 16835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "078cf354c5584dd9af7e6cb74a6376a8"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "da5d00647314def1e5eab6e93eb0ec90"
server_key_iv: "f39f5b33d7517efa1de823a391919863"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.431631 16835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "078cf354c5584dd9af7e6cb74a6376a8"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "da5d00647314def1e5eab6e93eb0ec90"
server_key_iv: "f39f5b33d7517efa1de823a391919863"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.433144 16835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:46:47.434053 16851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:47.434340 16835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:46:47.434469 16835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "078cf354c5584dd9af7e6cb74a6376a8"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "da5d00647314def1e5eab6e93eb0ec90"
server_key_iv: "f39f5b33d7517efa1de823a391919863"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.434561 16835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:47.449210 16835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:47.451882 16835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:47.452076 16835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:47.452346 16835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:47.452706 16835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:47.452765 16835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:47.452811 16835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:47.452842 16835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:47.460311 16835 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:41269
I20260807 08:46:47.460371 16973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:41269 every 8 connection(s)
I20260807 08:46:47.460762 16835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:46:47.465997 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 16835
I20260807 08:46:47.466149 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:46:47.466544 29106 external_mini_cluster.cc:1428] Setting key f0772a4e593ef4dbcfc09cc3149ac6ba
Aug 07 08:46:47 dist-test-slave-9zdj krb5kdc[16424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092407, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:47.472235 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.461594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:33477 (local address 127.28.108.190:41917)
0807 08:46:47.461705 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:47.461708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:47.461872 (+ 164us) server_negotiation.cc:407] Connection header received
0807 08:46:47.462454 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:47.462455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:47.462467 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:47.462528 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:47.463654 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.464179 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.465441 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.465779 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.465866 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.468857 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:47.468859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:47.468859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:47.468867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:47.470615 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:47.471159 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:47.471159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:47.471160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:47.471224 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:47.471490 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:47.471491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:47.471491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:47.471643 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:47.471706 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:47.472069 (+ 363us) server_negotiation.cc:299] Negotiation successful
0807 08:46:47.472156 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 08:46:47.472585 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.461518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41917 (local address 127.28.108.131:33477)
0807 08:46:47.461711 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:47.461752 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:47.462302 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:47.462587 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:47.462590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:47.463326 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:47.463542 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.463547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.464300 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.464301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.465183 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.465186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.465910 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.465911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.466105 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.466793 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:47.466805 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:47.468738 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:47.470732 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:47.470741 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:47.470744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:47.471071 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:47.471297 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:47.471298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:47.471298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:47.471422 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:47.471718 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:47.471724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:47.471923 (+ 199us) client_negotiation.cc:769] Sending connection context
0807 08:46:47.472038 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 08:46:47.472210 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260807 08:46:47.473420 16974 heartbeater.cc:347] Connected to a master server at 127.28.108.190:41917
I20260807 08:46:47.473537 16974 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:47.473817 16974 heartbeater.cc:511] Master 127.28.108.190:41917 requested a full tablet report, sending...
I20260807 08:46:47.474339 16474 ts_manager.cc:194] Registered new tserver with Master: 078cf354c5584dd9af7e6cb74a6376a8 (127.28.108.131:41269)
I20260807 08:46:47.474795 16474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:33477
I20260807 08:46:47.478916 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:46:47.488848 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.480739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.28.108.190:41917)
0807 08:46:47.480802 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:47.480804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:47.480856 (+ 52us) server_negotiation.cc:407] Connection header received
0807 08:46:47.480911 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:47.480911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:47.480927 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:47.480992 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:47.481850 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.482592 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.483546 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.483778 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.483849 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.485508 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:47.485509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:47.485509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:47.485517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:47.487116 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:47.487591 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:47.487592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:47.487592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:47.487655 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:47.488037 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:47.488038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:47.488038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:47.488339 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:47.488447 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:47.488691 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 08:46:47.488773 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 08:46:47.491498 16474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:46:47.493300 16474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:46:47.501592 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.497052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.28.108.130:41127)
0807 08:46:47.497425 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:47.497430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:47.497765 (+ 335us) server_negotiation.cc:407] Connection header received
0807 08:46:47.497912 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:47.497915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:47.497932 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:47.498021 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:47.498164 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.499220 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.500375 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.501103 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.501201 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.501267 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:47.501430 (+ 163us) server_negotiation.cc:299] Negotiation successful
0807 08:46:47.501480 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":126,"threads_started":1}
I20260807 08:46:47.501879 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.496656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.28.108.129:40735)
0807 08:46:47.496879 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:47.496885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:47.497155 (+ 270us) server_negotiation.cc:407] Connection header received
0807 08:46:47.497288 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:47.497291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:47.497307 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:47.497389 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:47.497728 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.498960 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.500630 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.501336 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.501447 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.501503 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:47.501728 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 08:46:47.501790 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260807 08:46:47.501627 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.496776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41127 (local address 127.0.0.1:44192)
0807 08:46:47.497690 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:47.497702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:47.497798 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:47.498037 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:47.498038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:47.498041 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:47.498085 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.498086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.499323 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.499324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.500269 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.500272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.501207 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.501208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.501286 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.501302 (+ 16us) client_negotiation.cc:769] Sending connection context
0807 08:46:47.501392 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 08:46:47.501495 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":90,"threads_started":1}
I20260807 08:46:47.502347 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.496528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:40735 (local address 127.0.0.1:51070)
0807 08:46:47.497028 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:47.497074 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:47.497183 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:47.497497 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:47.497500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:47.497503 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:47.497598 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.497601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.499095 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.499096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.500526 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.500529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.501473 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.501474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.501536 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.501540 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:46:47.501601 (+ 61us) client_negotiation.cc:240] Negotiation successful
0807 08:46:47.501672 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":38,"thread_start_us":106,"threads_started":1}
I20260807 08:46:47.502840 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.496527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:41269 (local address 127.0.0.1:36882)
0807 08:46:47.497030 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:47.497077 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:47.497177 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:47.497553 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:47.497554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:47.497557 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:47.497647 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.497649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.498902 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.498903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.500528 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.500529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.502578 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.502579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.502644 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.502649 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:46:47.502718 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 08:46:47.502766 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":38,"thread_start_us":76,"threads_started":1}
I20260807 08:46:47.502952 16760 tablet_service.cc:1467] Processing CreateTablet for tablet 5a2a938d6287449ea0aaa4e88909a5b1 (DEFAULT_TABLE table=test-table [id=1dd6c1615b214f0da5ee579a98663ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:47.503224 16610 tablet_service.cc:1467] Processing CreateTablet for tablet 5a2a938d6287449ea0aaa4e88909a5b1 (DEFAULT_TABLE table=test-table [id=1dd6c1615b214f0da5ee579a98663ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:47.503400 16760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a2a938d6287449ea0aaa4e88909a5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:47.503607 16610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a2a938d6287449ea0aaa4e88909a5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:47.505946 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.496684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.28.108.131:41269)
0807 08:46:47.496959 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:47.496964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:47.497282 (+ 318us) server_negotiation.cc:407] Connection header received
0807 08:46:47.497344 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:47.497347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:47.497368 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:47.497452 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:47.497748 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.498769 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.501547 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.502409 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.502487 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.502541 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:47.505746 (+ 3205us) server_negotiation.cc:299] Negotiation successful
0807 08:46:47.505833 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":187,"threads_started":1}
I20260807 08:46:47.506789 16903 tablet_service.cc:1467] Processing CreateTablet for tablet 5a2a938d6287449ea0aaa4e88909a5b1 (DEFAULT_TABLE table=test-table [id=1dd6c1615b214f0da5ee579a98663ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:47.506932 16993 tablet_bootstrap.cc:492] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: Bootstrap starting.
I20260807 08:46:47.507194 16903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a2a938d6287449ea0aaa4e88909a5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:47.507405 16994 tablet_bootstrap.cc:492] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147: Bootstrap starting.
I20260807 08:46:47.507767 16993 tablet_bootstrap.cc:654] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:47.508152 16994 tablet_bootstrap.cc:654] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:47.508267 16993 log.cc:824] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:47.508793 16994 log.cc:824] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:47.509418 16993 tablet_bootstrap.cc:492] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: No bootstrap required, opened a new log
I20260807 08:46:47.509479 16993 ts_tablet_manager.cc:1397] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:46:47.509713 16994 tablet_bootstrap.cc:492] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147: No bootstrap required, opened a new log
I20260807 08:46:47.509779 16994 ts_tablet_manager.cc:1397] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:46:47.511179 16997 tablet_bootstrap.cc:492] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8: Bootstrap starting.
I20260807 08:46:47.511245 16993 raft_consensus.cc:348] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73404b6f29194826bdcbdb7a36418147" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40735 } } peers { permanent_uuid: "078cf354c5584dd9af7e6cb74a6376a8" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 41269 } } peers { permanent_uuid: "d502af8ad8224d68a811e6f66f57e432" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41127 } }
I20260807 08:46:47.511463 16993 raft_consensus.cc:374] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:47.511519 16993 raft_consensus.cc:729] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502af8ad8224d68a811e6f66f57e432, State: Initialized, Role: FOLLOWER
I20260807 08:46:47.511683 16993 consensus_queue.cc:260] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73404b6f29194826bdcbdb7a36418147" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40735 } } peers { permanent_uuid: "078cf354c5584dd9af7e6cb74a6376a8" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 41269 } } peers { permanent_uuid: "d502af8ad8224d68a811e6f66f57e432" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41127 } }
I20260807 08:46:47.512058 16993 ts_tablet_manager.cc:1428] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:46:47.512068 16997 tablet_bootstrap.cc:654] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:47.512027 16994 raft_consensus.cc:348] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73404b6f29194826bdcbdb7a36418147" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40735 } } peers { permanent_uuid: "078cf354c5584dd9af7e6cb74a6376a8" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 41269 } } peers { permanent_uuid: "d502af8ad8224d68a811e6f66f57e432" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41127 } }
I20260807 08:46:47.512260 16994 raft_consensus.cc:374] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:47.512283 16830 heartbeater.cc:503] Master 127.28.108.190:41917 was elected leader, sending a full tablet report...
I20260807 08:46:47.512310 16994 raft_consensus.cc:729] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73404b6f29194826bdcbdb7a36418147, State: Initialized, Role: FOLLOWER
I20260807 08:46:47.512435 16994 consensus_queue.cc:260] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73404b6f29194826bdcbdb7a36418147" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40735 } } peers { permanent_uuid: "078cf354c5584dd9af7e6cb74a6376a8" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 41269 } } peers { permanent_uuid: "d502af8ad8224d68a811e6f66f57e432" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41127 } }
I20260807 08:46:47.512740 16994 ts_tablet_manager.cc:1428] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 08:46:47.512813 16997 log.cc:824] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:47.512871 16682 heartbeater.cc:503] Master 127.28.108.190:41917 was elected leader, sending a full tablet report...
I20260807 08:46:47.513875 16997 tablet_bootstrap.cc:492] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8: No bootstrap required, opened a new log
I20260807 08:46:47.513990 16997 ts_tablet_manager.cc:1397] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:46:47.515568 16997 raft_consensus.cc:348] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73404b6f29194826bdcbdb7a36418147" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40735 } } peers { permanent_uuid: "078cf354c5584dd9af7e6cb74a6376a8" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 41269 } } peers { permanent_uuid: "d502af8ad8224d68a811e6f66f57e432" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41127 } }
I20260807 08:46:47.515738 16997 raft_consensus.cc:374] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:47.515765 16997 raft_consensus.cc:729] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078cf354c5584dd9af7e6cb74a6376a8, State: Initialized, Role: FOLLOWER
I20260807 08:46:47.515911 16997 consensus_queue.cc:260] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73404b6f29194826bdcbdb7a36418147" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40735 } } peers { permanent_uuid: "078cf354c5584dd9af7e6cb74a6376a8" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 41269 } } peers { permanent_uuid: "d502af8ad8224d68a811e6f66f57e432" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41127 } }
I20260807 08:46:47.516225 16997 ts_tablet_manager.cc:1428] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:46:47.516403 16974 heartbeater.cc:503] Master 127.28.108.190:41917 was elected leader, sending a full tablet report...
W20260807 08:46:47.531787 16831 tablet.cc:2369] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:46:47.587455 16683 tablet.cc:2369] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:46:47.607725 17001 raft_consensus.cc:482] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:47.607882 17001 raft_consensus.cc:504] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73404b6f29194826bdcbdb7a36418147" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40735 } } peers { permanent_uuid: "078cf354c5584dd9af7e6cb74a6376a8" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 41269 } } peers { permanent_uuid: "d502af8ad8224d68a811e6f66f57e432" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41127 } }
I20260807 08:46:47.608464 17001 leader_election.cc:288] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73404b6f29194826bdcbdb7a36418147 (127.28.108.129:40735), d502af8ad8224d68a811e6f66f57e432 (127.28.108.130:41127)
I20260807 08:46:47.613113 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.608892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:50015 (local address 127.28.108.130:41127)
0807 08:46:47.608937 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:47.608939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:47.609175 (+ 236us) server_negotiation.cc:407] Connection header received
0807 08:46:47.609206 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:47.609207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:47.609217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:47.609271 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:47.609529 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.610231 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.611942 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.612542 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.612647 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.612656 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:47.612977 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 08:46:47.613034 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:46:47.613113 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.608878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:54311 (local address 127.28.108.129:40735)
0807 08:46:47.608933 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:47.608935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:47.608947 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:46:47.608982 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:47.608983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:47.609011 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:47.609092 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:47.609314 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.610131 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.611943 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.612542 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.612646 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.612656 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:47.612977 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 08:46:47.613034 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 08:46:47.613090 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.608728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:40735 (local address 127.28.108.131:54311)
0807 08:46:47.608802 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:47.608815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:47.608871 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:47.609129 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:47.609130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:47.609132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:47.609222 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.609223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.610258 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.610259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.611776 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.611780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.612684 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.612685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.612845 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.612848 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:46:47.612936 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 08:46:47.612975 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":68}
I20260807 08:46:47.613101 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.608729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:41127 (local address 127.28.108.131:50015)
0807 08:46:47.609034 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:47.609049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:47.609128 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:47.609293 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:47.609294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:47.609296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:47.609435 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.609437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.610321 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.610322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.611818 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:47.611820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.612684 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.612685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:47.612797 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.612803 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:46:47.612887 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 08:46:47.612955 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":35,"thread_start_us":199,"threads_started":1}
I20260807 08:46:47.613761 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2a938d6287449ea0aaa4e88909a5b1" candidate_uuid: "078cf354c5584dd9af7e6cb74a6376a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502af8ad8224d68a811e6f66f57e432" is_pre_election: true
I20260807 08:46:47.613770 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2a938d6287449ea0aaa4e88909a5b1" candidate_uuid: "078cf354c5584dd9af7e6cb74a6376a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73404b6f29194826bdcbdb7a36418147" is_pre_election: true
I20260807 08:46:47.613956 16781 raft_consensus.cc:2454] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078cf354c5584dd9af7e6cb74a6376a8 in term 0.
I20260807 08:46:47.613977 16632 raft_consensus.cc:2454] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078cf354c5584dd9af7e6cb74a6376a8 in term 0.
I20260807 08:46:47.614254 16856 leader_election.cc:302] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 078cf354c5584dd9af7e6cb74a6376a8, d502af8ad8224d68a811e6f66f57e432; no voters:
I20260807 08:46:47.614401 17001 raft_consensus.cc:2781] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:46:47.614459 17001 raft_consensus.cc:482] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:46:47.614517 17001 raft_consensus.cc:3037] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:47.615500 17001 raft_consensus.cc:504] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73404b6f29194826bdcbdb7a36418147" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40735 } } peers { permanent_uuid: "078cf354c5584dd9af7e6cb74a6376a8" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 41269 } } peers { permanent_uuid: "d502af8ad8224d68a811e6f66f57e432" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41127 } }
I20260807 08:46:47.615655 17001 leader_election.cc:288] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [CANDIDATE]: Term 1 election: Requested vote from peers 73404b6f29194826bdcbdb7a36418147 (127.28.108.129:40735), d502af8ad8224d68a811e6f66f57e432 (127.28.108.130:41127)
I20260807 08:46:47.615873 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2a938d6287449ea0aaa4e88909a5b1" candidate_uuid: "078cf354c5584dd9af7e6cb74a6376a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73404b6f29194826bdcbdb7a36418147"
I20260807 08:46:47.615896 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2a938d6287449ea0aaa4e88909a5b1" candidate_uuid: "078cf354c5584dd9af7e6cb74a6376a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502af8ad8224d68a811e6f66f57e432"
I20260807 08:46:47.615969 16781 raft_consensus.cc:3037] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:47.615983 16632 raft_consensus.cc:3037] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:47.616814 16781 raft_consensus.cc:2454] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078cf354c5584dd9af7e6cb74a6376a8 in term 1.
I20260807 08:46:47.616838 16632 raft_consensus.cc:2454] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078cf354c5584dd9af7e6cb74a6376a8 in term 1.
I20260807 08:46:47.617038 16852 leader_election.cc:302] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 078cf354c5584dd9af7e6cb74a6376a8, 73404b6f29194826bdcbdb7a36418147; no voters:
I20260807 08:46:47.617165 17001 raft_consensus.cc:2781] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:47.617358 17001 raft_consensus.cc:686] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 1 LEADER]: Becoming Leader. State: Replica: 078cf354c5584dd9af7e6cb74a6376a8, State: Running, Role: LEADER
I20260807 08:46:47.617470 17001 consensus_queue.cc:237] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73404b6f29194826bdcbdb7a36418147" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40735 } } peers { permanent_uuid: "078cf354c5584dd9af7e6cb74a6376a8" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 41269 } } peers { permanent_uuid: "d502af8ad8224d68a811e6f66f57e432" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41127 } }
I20260807 08:46:47.618494 16473 catalog_manager.cc:5495] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 078cf354c5584dd9af7e6cb74a6376a8 (127.28.108.131). New cstate: current_term: 1 leader_uuid: "078cf354c5584dd9af7e6cb74a6376a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73404b6f29194826bdcbdb7a36418147" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078cf354c5584dd9af7e6cb74a6376a8" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 41269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502af8ad8224d68a811e6f66f57e432" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41127 } health_report { overall_health: UNKNOWN } } }
I20260807 08:46:47.650216 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.647600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.28.108.131:41269)
0807 08:46:47.647667 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:47.647669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:47.647685 (+ 16us) server_negotiation.cc:407] Connection header received
0807 08:46:47.647724 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:47.647724 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:47.647748 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:47.647881 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:46:47.648006 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.648539 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.649332 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.649542 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.649670 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.649841 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:46:47.649946 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:46:47.650062 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:47.650094 (+ 32us) server_negotiation.cc:299] Negotiation successful
0807 08:46:47.650105 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 08:46:47.652356 16632 raft_consensus.cc:1261] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147 [term 1 FOLLOWER]: Refusing update from remote peer 078cf354c5584dd9af7e6cb74a6376a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:47.652360 16781 raft_consensus.cc:1261] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432 [term 1 FOLLOWER]: Refusing update from remote peer 078cf354c5584dd9af7e6cb74a6376a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:47.652770 17004 consensus_queue.cc:1035] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73404b6f29194826bdcbdb7a36418147" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:47.652933 17001 consensus_queue.cc:1035] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502af8ad8224d68a811e6f66f57e432" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 41127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:47.656898 17009 mvcc.cc:205] Tried to move back new op lower bound from 7315834501740867584 to 7315834501602013184. Current Snapshot: MvccSnapshot[applied={T|T < 7315834501740867584}]
I20260807 08:46:47.658499 16473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:47.658638 16473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:47.659750 16473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ca6a5a1d71048fc953fc0e772382dba: Sending DeleteTablet for 3 replicas of tablet 5a2a938d6287449ea0aaa4e88909a5b1
I20260807 08:46:47.660166 16610 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a2a938d6287449ea0aaa4e88909a5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:47 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:51070
I20260807 08:46:47.660322 16760 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a2a938d6287449ea0aaa4e88909a5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:47 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:44192
I20260807 08:46:47.660455 17017 tablet_replica.cc:330] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147: stopping tablet replica
I20260807 08:46:47.660575 17017 raft_consensus.cc:2229] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:47.660652 17019 tablet_replica.cc:330] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: stopping tablet replica
I20260807 08:46:47.660701 17017 raft_consensus.cc:2258] T 5a2a938d6287449ea0aaa4e88909a5b1 P 73404b6f29194826bdcbdb7a36418147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:47.660637 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 16543
I20260807 08:46:47.661262 16903 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a2a938d6287449ea0aaa4e88909a5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:47 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:36882
I20260807 08:46:47.660768 17019 raft_consensus.cc:2229] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:47.664834 17019 raft_consensus.cc:2258] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:47.665086 17020 tablet_replica.cc:330] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8: stopping tablet replica
I20260807 08:46:47.665324 17020 raft_consensus.cc:2229] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:46:47.665548 17020 raft_consensus.cc:2258] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:47.666313 17020 ts_tablet_manager.cc:1905] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:47.668133 17019 ts_tablet_manager.cc:1905] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:47.670512 17019 ts_tablet_manager.cc:1918] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:47.670614 17019 log.cc:1196] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/5a2a938d6287449ea0aaa4e88909a5b1
I20260807 08:46:47.670866 17020 ts_tablet_manager.cc:1918] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:47.670959 17020 log.cc:1196] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/5a2a938d6287449ea0aaa4e88909a5b1
I20260807 08:46:47.670997 17019 ts_tablet_manager.cc:1939] T 5a2a938d6287449ea0aaa4e88909a5b1 P d502af8ad8224d68a811e6f66f57e432: Deleting consensus metadata
I20260807 08:46:47.671262 17020 ts_tablet_manager.cc:1939] T 5a2a938d6287449ea0aaa4e88909a5b1 P 078cf354c5584dd9af7e6cb74a6376a8: Deleting consensus metadata
I20260807 08:46:47.671723 16459 catalog_manager.cc:4841] TS d502af8ad8224d68a811e6f66f57e432 (127.28.108.130:41127): tablet 5a2a938d6287449ea0aaa4e88909a5b1 (table test-table [id=1dd6c1615b214f0da5ee579a98663ada]) successfully deleted
I20260807 08:46:47.671875 16459 catalog_manager.cc:4841] TS 078cf354c5584dd9af7e6cb74a6376a8 (127.28.108.131:41269): tablet 5a2a938d6287449ea0aaa4e88909a5b1 (table test-table [id=1dd6c1615b214f0da5ee579a98663ada]) successfully deleted
W20260807 08:46:47.672843 16456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:40735 (error 108)
I20260807 08:46:47.673488 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.673328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:40735 (local address 127.0.0.1:51074)
0807 08:46:47.673401 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:47.673450 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 08:46:47.673678 16456 catalog_manager.cc:4584] TS 73404b6f29194826bdcbdb7a36418147 (127.28.108.129:40735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a2a938d6287449ea0aaa4e88909a5b1: Network error: Client connection negotiation failed: client connection to 127.28.108.129:40735: connect: Connection refused (error 111)
I20260807 08:46:47.673740 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 16688
I20260807 08:46:47.684922 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 16835
I20260807 08:46:47.694764 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 16438
2026-08-07T08:46:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2311 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T08:46:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:46:47Z Disabled control of system clock
I20260807 08:46:47.728941 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:33675
--webserver_interface=127.28.108.190
--webserver_port=0
--builtin_ntp_servers=127.28.108.148:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:33675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 08:46:47.819185 17025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:47.819459 17025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:47.819486 17025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:47.820729 17025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:47.820775 17025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:47.820788 17025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:47.820802 17025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:47.820814 17025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:47.822391 17025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:33675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:33675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:47.822751 17025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:47.823045 17025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:47.829322 17033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:47.829342 17032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:47.829533 17025 server_base.cc:1034] running on GCE node
W20260807 08:46:47.829375 17035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:47.830114 17025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:47.830565 17025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:47.831743 17025 hybrid_clock.cc:648] HybridClock initialized: now 1786092407831716 us; error 38 us; skew 500 ppm
I20260807 08:46:47.833395 17025 webserver.cc:460] Webserver started at http://127.28.108.190:36783/ using document root <none> and password file <none>
I20260807 08:46:47.833782 17025 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:47.833884 17025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:47.834074 17025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:47.835412 17025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "0e0db6836ced46f78883513df044f13f"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "2e630a9e7149c000c84a9d2ca9bf8b5d"
server_key_iv: "9abcc3a5e1f50b7876f652e9bdc2ad5e"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.835861 17025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "0e0db6836ced46f78883513df044f13f"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "2e630a9e7149c000c84a9d2ca9bf8b5d"
server_key_iv: "9abcc3a5e1f50b7876f652e9bdc2ad5e"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.837525 17025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 08:46:47.838644 17043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:47.838934 17025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:47.839067 17025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "0e0db6836ced46f78883513df044f13f"
format_stamp: "Formatted at 2026-08-07 08:46:47 on dist-test-slave-9zdj"
server_key: "2e630a9e7149c000c84a9d2ca9bf8b5d"
server_key_iv: "9abcc3a5e1f50b7876f652e9bdc2ad5e"
server_key_version: "encryptionkey@0"
I20260807 08:46:47.839164 17025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:47.870504 17025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:47.871058 17025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:47.871290 17025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:47.876556 17025 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:33675
I20260807 08:46:47.876608 17105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:33675 every 8 connection(s)
I20260807 08:46:47.877043 17025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:47.878400 17106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:47.881028 17106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f: Bootstrap starting.
I20260807 08:46:47.881759 17106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:47.882107 17106 log.cc:824] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:47.883020 17106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f: No bootstrap required, opened a new log
I20260807 08:46:47.885056 17106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0db6836ced46f78883513df044f13f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33675 } }
I20260807 08:46:47.885265 17106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:47.885293 17106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0db6836ced46f78883513df044f13f, State: Initialized, Role: FOLLOWER
I20260807 08:46:47.885445 17106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0db6836ced46f78883513df044f13f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33675 } }
I20260807 08:46:47.885514 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 17025
I20260807 08:46:47.885553 17106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:47.885594 17106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:47.885660 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:46:47.885670 17106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:47.885962 29106 external_mini_cluster.cc:1428] Setting key 044920b45b63ea2ae260b7068395a177
I20260807 08:46:47.886354 17106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0db6836ced46f78883513df044f13f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33675 } }
I20260807 08:46:47.886485 17106 leader_election.cc:302] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e0db6836ced46f78883513df044f13f; no voters:
I20260807 08:46:47.886683 17106 leader_election.cc:288] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:46:47.886755 17112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:47.886929 17112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [term 1 LEADER]: Becoming Leader. State: Replica: 0e0db6836ced46f78883513df044f13f, State: Running, Role: LEADER
I20260807 08:46:47.886974 17106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:47.887066 17112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0db6836ced46f78883513df044f13f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33675 } }
I20260807 08:46:47.887458 17112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e0db6836ced46f78883513df044f13f. Latest consensus state: current_term: 1 leader_uuid: "0e0db6836ced46f78883513df044f13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0db6836ced46f78883513df044f13f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33675 } } }
I20260807 08:46:47.887538 17112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:47.887946 17114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e0db6836ced46f78883513df044f13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0db6836ced46f78883513df044f13f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 33675 } } }
I20260807 08:46:47.888054 17114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:47.888229 17123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:47.888957 17123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:47.891376 17123 catalog_manager.cc:1347] Generated new cluster ID: d58dc67e2d6440aba8ce659d8212df6f
I20260807 08:46:47.891448 17123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:46:47.893078 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:47.886594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.28.108.190:33675)
0807 08:46:47.886873 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:47.886883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:47.886919 (+ 36us) server_negotiation.cc:407] Connection header received
0807 08:46:47.887110 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:47.887120 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:47.887207 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:47.887670 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:47.887806 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.889506 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.890758 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:47.891365 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:47.891506 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:47.892320 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:47.892324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:47.892324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:47.892521 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:47.892564 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:47.892576 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:47.892651 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:47.892756 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 08:46:47.892849 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":97,"threads_started":1}
I20260807 08:46:47.911064 17123 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:46:47.911840 17123 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:47.919167 17123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f: Generated new TSK 0
I20260807 08:46:47.919492 17123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:46:47.929144 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33675
--builtin_ntp_servers=127.28.108.148:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 08:46:48.023142 17137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:48.023434 17137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:48.023476 17137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:48.025249 17137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:48.025328 17137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:48.025365 17137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:46:48.027231 17137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:48.027580 17137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:48.027987 17137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:48.034190 17143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:48.034317 17146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:48.034192 17144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:48.035120 17137 server_base.cc:1034] running on GCE node
I20260807 08:46:48.035537 17137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:48.035887 17137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:48.037129 17137 hybrid_clock.cc:648] HybridClock initialized: now 1786092408037096 us; error 52 us; skew 500 ppm
I20260807 08:46:48.038887 17137 webserver.cc:460] Webserver started at http://127.28.108.129:35663/ using document root <none> and password file <none>
I20260807 08:46:48.039232 17137 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:48.039316 17137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:48.039482 17137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:48.040683 17137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "df5cac93a1294a949962f4418bf51e66"
format_stamp: "Formatted at 2026-08-07 08:46:48 on dist-test-slave-9zdj"
server_key: "ea49694f1613556bff0d3ac26921c1f7"
server_key_iv: "7b073b3f0132dc980f2ec9324e0e585e"
server_key_version: "encryptionkey@0"
I20260807 08:46:48.041103 17137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "df5cac93a1294a949962f4418bf51e66"
format_stamp: "Formatted at 2026-08-07 08:46:48 on dist-test-slave-9zdj"
server_key: "ea49694f1613556bff0d3ac26921c1f7"
server_key_iv: "7b073b3f0132dc980f2ec9324e0e585e"
server_key_version: "encryptionkey@0"
I20260807 08:46:48.042856 17137 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 08:46:48.044073 17153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:48.044476 17137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 08:46:48.044617 17137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "df5cac93a1294a949962f4418bf51e66"
format_stamp: "Formatted at 2026-08-07 08:46:48 on dist-test-slave-9zdj"
server_key: "ea49694f1613556bff0d3ac26921c1f7"
server_key_iv: "7b073b3f0132dc980f2ec9324e0e585e"
server_key_version: "encryptionkey@0"
I20260807 08:46:48.044718 17137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:48.062150 17137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:48.062608 17137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:48.062812 17137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:48.063118 17137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:48.063526 17137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:48.063591 17137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:48.063630 17137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:48.063678 17137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:48.071460 17137 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:40435
I20260807 08:46:48.071574 17279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:40435 every 8 connection(s)
I20260807 08:46:48.071995 17137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:46:48.076422 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 17137
I20260807 08:46:48.076548 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:46:48.076834 29106 external_mini_cluster.cc:1428] Setting key c06343653c397f41d52710e8430bebdd
I20260807 08:46:48.078363 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33675
--builtin_ntp_servers=127.28.108.148:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 08:46:48.079674 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.072972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:53529 (local address 127.28.108.190:33675)
0807 08:46:48.073054 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:48.073057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:48.073419 (+ 362us) server_negotiation.cc:407] Connection header received
0807 08:46:48.074171 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:48.074173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:48.074188 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:48.074262 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:48.075337 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.075889 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.077491 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.077883 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.077980 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.079303 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:48.079305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:48.079305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:48.079314 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:48.079331 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:48.079336 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:48.079399 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:48.079554 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 08:46:48.079596 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 08:46:48.080057 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.072966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33675 (local address 127.28.108.129:53529)
0807 08:46:48.073194 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:48.073277 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:48.074016 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:48.074483 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:48.074488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:48.074498 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:48.075077 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.075083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.076007 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.076008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.077399 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.077403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.078038 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.078040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.078269 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.079059 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:46:48.079060 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:48.079112 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:46:48.079150 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:48.079442 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:48.079451 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:48.079455 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:46:48.079550 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 08:46:48.079708 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
I20260807 08:46:48.080869 17280 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33675
I20260807 08:46:48.081027 17280 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:48.081300 17280 heartbeater.cc:511] Master 127.28.108.190:33675 requested a full tablet report, sending...
I20260807 08:46:48.082089 17060 ts_manager.cc:194] Registered new tserver with Master: df5cac93a1294a949962f4418bf51e66 (127.28.108.129:40435)
I20260807 08:46:48.082876 17060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.129:53529
W20260807 08:46:48.178054 17283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:48.178378 17283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:48.178427 17283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:48.179910 17283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:48.180017 17283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:48.180071 17283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:46:48.181901 17283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:48.182282 17283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:48.182667 17283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:48.188928 17289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:48.188953 17288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:48.188930 17292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:48.189239 17283 server_base.cc:1034] running on GCE node
I20260807 08:46:48.189498 17283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:48.189939 17283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:48.191159 17283 hybrid_clock.cc:648] HybridClock initialized: now 1786092408191147 us; error 34 us; skew 500 ppm
I20260807 08:46:48.192704 17283 webserver.cc:460] Webserver started at http://127.28.108.130:39963/ using document root <none> and password file <none>
I20260807 08:46:48.193025 17283 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:48.193115 17283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:48.193320 17283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:48.194845 17283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "f3096bb809334cf0a25ce299d04a99c4"
format_stamp: "Formatted at 2026-08-07 08:46:48 on dist-test-slave-9zdj"
server_key: "ca176117e216be1211ac1ebdf319eeca"
server_key_iv: "c4acab03a4f8aa63929f6fb372ad05d4"
server_key_version: "encryptionkey@0"
I20260807 08:46:48.195284 17283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "f3096bb809334cf0a25ce299d04a99c4"
format_stamp: "Formatted at 2026-08-07 08:46:48 on dist-test-slave-9zdj"
server_key: "ca176117e216be1211ac1ebdf319eeca"
server_key_iv: "c4acab03a4f8aa63929f6fb372ad05d4"
server_key_version: "encryptionkey@0"
I20260807 08:46:48.197027 17283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260807 08:46:48.198254 17299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:48.198557 17283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:46:48.198712 17283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "f3096bb809334cf0a25ce299d04a99c4"
format_stamp: "Formatted at 2026-08-07 08:46:48 on dist-test-slave-9zdj"
server_key: "ca176117e216be1211ac1ebdf319eeca"
server_key_iv: "c4acab03a4f8aa63929f6fb372ad05d4"
server_key_version: "encryptionkey@0"
I20260807 08:46:48.198812 17283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:48.211364 17283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:48.211833 17283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:48.212052 17283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:48.212471 17283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:48.212924 17283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:48.212991 17283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:48.213038 17283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:48.213073 17283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:48.220758 17283 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:39533
I20260807 08:46:48.220803 17424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:39533 every 8 connection(s)
I20260807 08:46:48.221235 17283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:46:48.227077 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 17283
I20260807 08:46:48.227223 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:46:48.227468 29106 external_mini_cluster.cc:1428] Setting key e03d4b3dc83c94383b863497d933c4e0
I20260807 08:46:48.228543 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.222401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:44135 (local address 127.28.108.190:33675)
0807 08:46:48.222495 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:48.222497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:48.222899 (+ 402us) server_negotiation.cc:407] Connection header received
0807 08:46:48.223610 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:48.223612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:48.223626 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:48.223707 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:48.224516 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.225134 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.226371 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.226602 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.226705 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.228210 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:48.228212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:48.228213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:48.228218 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:48.228239 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:48.228244 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:48.228289 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:48.228409 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 08:46:48.228451 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 08:46:48.228839 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.222469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33675 (local address 127.28.108.130:44135)
0807 08:46:48.222734 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:48.222776 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:48.223475 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:48.223793 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:48.223797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:48.223806 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:48.224405 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.224411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.225262 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.225263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.226261 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.226264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.226733 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.226734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.226961 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.228001 (+ 1040us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:46:48.228002 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:48.228056 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:46:48.228087 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:48.228302 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:48.228312 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:48.228316 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:46:48.228419 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 08:46:48.228564 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260807 08:46:48.228988 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33675
--builtin_ntp_servers=127.28.108.148:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 08:46:48.229398 17425 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33675
I20260807 08:46:48.229528 17425 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:48.229838 17425 heartbeater.cc:511] Master 127.28.108.190:33675 requested a full tablet report, sending...
I20260807 08:46:48.230544 17060 ts_manager.cc:194] Registered new tserver with Master: f3096bb809334cf0a25ce299d04a99c4 (127.28.108.130:39533)
I20260807 08:46:48.230996 17060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.130:44135
W20260807 08:46:48.322074 17428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:48.322471 17428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:48.322506 17428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:48.323854 17428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:48.323908 17428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:48.323938 17428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:46:48.325459 17428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:48.325793 17428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:48.326188 17428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:48.332124 17434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:48.332134 17433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:48.332496 17438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:48.332623 17428 server_base.cc:1034] running on GCE node
I20260807 08:46:48.332887 17428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:48.333168 17428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:48.334455 17428 hybrid_clock.cc:648] HybridClock initialized: now 1786092408334438 us; error 29 us; skew 500 ppm
I20260807 08:46:48.336004 17428 webserver.cc:460] Webserver started at http://127.28.108.131:35671/ using document root <none> and password file <none>
I20260807 08:46:48.336320 17428 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:48.336416 17428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:48.336613 17428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:48.337873 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "baf95a85111248a08fdcaa397c523c07"
format_stamp: "Formatted at 2026-08-07 08:46:48 on dist-test-slave-9zdj"
server_key: "a206c8be420c05c2e166c84f05c5b821"
server_key_iv: "dd5e27a1780bd59ed81c869bb2431657"
server_key_version: "encryptionkey@0"
I20260807 08:46:48.338284 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "baf95a85111248a08fdcaa397c523c07"
format_stamp: "Formatted at 2026-08-07 08:46:48 on dist-test-slave-9zdj"
server_key: "a206c8be420c05c2e166c84f05c5b821"
server_key_iv: "dd5e27a1780bd59ed81c869bb2431657"
server_key_version: "encryptionkey@0"
I20260807 08:46:48.339813 17428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:46:48.340878 17444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:48.341332 17428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:48.341460 17428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "baf95a85111248a08fdcaa397c523c07"
format_stamp: "Formatted at 2026-08-07 08:46:48 on dist-test-slave-9zdj"
server_key: "a206c8be420c05c2e166c84f05c5b821"
server_key_iv: "dd5e27a1780bd59ed81c869bb2431657"
server_key_version: "encryptionkey@0"
I20260807 08:46:48.341562 17428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:48.372128 17428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:48.372697 17428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:48.372895 17428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:48.373234 17428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:48.373724 17428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:48.373791 17428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:48.373835 17428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:48.373850 17428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:48.381402 17428 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:34747
I20260807 08:46:48.381443 17565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:34747 every 8 connection(s)
I20260807 08:46:48.381945 17428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:46:48.387110 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 17428
I20260807 08:46:48.387276 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:46:48.387539 29106 external_mini_cluster.cc:1428] Setting key 882ce29468262fe8cb4ce2652fef920b
I20260807 08:46:48.389192 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.382935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:40203 (local address 127.28.108.190:33675)
0807 08:46:48.383104 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:48.383106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:48.383369 (+ 263us) server_negotiation.cc:407] Connection header received
0807 08:46:48.383937 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:48.383939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:48.383950 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:48.384017 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:48.384904 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.385573 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.386983 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.387407 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.387534 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.388940 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:48.388942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:48.388942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:48.388947 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:48.388964 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:48.388968 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:48.389026 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:48.389083 (+ 57us) server_negotiation.cc:299] Negotiation successful
0807 08:46:48.389116 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260807 08:46:48.389439 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.382954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:33675 (local address 127.28.108.131:40203)
0807 08:46:48.383181 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:48.383235 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:48.383825 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:48.384080 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:48.384084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:48.384093 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:48.384798 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.384804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.385845 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.385846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.386833 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.386837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.387593 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.387595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.387834 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.388751 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:46:48.388752 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:48.388804 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:46:48.388838 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:48.389029 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:48.389032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:48.389034 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 08:46:48.389084 (+ 50us) client_negotiation.cc:240] Negotiation successful
0807 08:46:48.389185 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":114,"threads_started":1}
I20260807 08:46:48.390059 17566 heartbeater.cc:347] Connected to a master server at 127.28.108.190:33675
I20260807 08:46:48.390178 17566 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:48.390362 17566 heartbeater.cc:511] Master 127.28.108.190:33675 requested a full tablet report, sending...
I20260807 08:46:48.390764 17060 ts_manager.cc:194] Registered new tserver with Master: baf95a85111248a08fdcaa397c523c07 (127.28.108.131:34747)
I20260807 08:46:48.391147 17060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.131:40203
I20260807 08:46:48.399958 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:46:48.405540 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.401808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.28.108.190:33675)
0807 08:46:48.401886 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:48.401889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:48.401991 (+ 102us) server_negotiation.cc:407] Connection header received
0807 08:46:48.402108 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:48.402110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:48.402123 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:48.402179 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:48.402418 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.402986 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.403987 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.404196 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.404261 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.405077 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:48.405078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:48.405078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:48.405084 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:48.405101 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:48.405105 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:48.405224 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:48.405367 (+ 143us) server_negotiation.cc:299] Negotiation successful
0807 08:46:48.405406 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 08:46:48.407506 17060 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:46:48.409348 17060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:46:48.418001 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.412619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:40435 (local address 127.0.0.1:34808)
0807 08:46:48.412832 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:48.412842 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:48.412901 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:48.413678 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:48.413679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:48.413682 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:48.413727 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.413729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.415219 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.415220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.416391 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.416398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.417670 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.417671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.417745 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.417751 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:46:48.417803 (+ 52us) client_negotiation.cc:240] Negotiation successful
0807 08:46:48.417863 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":4,"thread_start_us":49,"threads_started":1}
I20260807 08:46:48.418001 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.412386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:39533 (local address 127.0.0.1:35736)
0807 08:46:48.412759 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:48.412770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:48.412826 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:48.413077 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:48.413081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:48.413084 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:48.413331 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.413403 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.414588 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.414590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.416081 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.416084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.417451 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.417452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.417569 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.417586 (+ 17us) client_negotiation.cc:769] Sending connection context
0807 08:46:48.417743 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 08:46:48.417861 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":23,"thread_start_us":76,"threads_started":1}
I20260807 08:46:48.418226 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.412381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:34747 (local address 127.0.0.1:44590)
0807 08:46:48.412519 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:48.412569 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:48.412648 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:48.413403 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:48.413404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:48.413407 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:48.413478 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.413481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.415167 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.415169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.416562 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.416568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.417884 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.417885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.417960 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.417966 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:46:48.418020 (+ 54us) client_negotiation.cc:240] Negotiation successful
0807 08:46:48.418077 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260807 08:46:48.418412 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.412470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.28.108.130:39533)
0807 08:46:48.412724 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:48.412730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:48.412865 (+ 135us) server_negotiation.cc:407] Connection header received
0807 08:46:48.412923 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:48.412926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:48.412947 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:48.413042 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:48.413520 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.414458 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.416208 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.417330 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.417431 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.417551 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:48.417790 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 08:46:48.418310 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":111,"threads_started":1}
I20260807 08:46:48.418686 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.412588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.28.108.131:34747)
0807 08:46:48.413033 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:48.413038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:48.413049 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:46:48.413089 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:48.413091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:48.413108 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:48.413314 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:48.413611 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.415049 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.416757 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.417746 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.418387 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.418513 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:48.418544 (+ 31us) server_negotiation.cc:299] Negotiation successful
0807 08:46:48.418592 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":104,"threads_started":1}
I20260807 08:46:48.419363 17495 tablet_service.cc:1467] Processing CreateTablet for tablet bdb19a28f07c4e3d80061d63a5817a81 (DEFAULT_TABLE table=test-table [id=f684b7dc1de94a36a7652b3b694101f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:48.419303 17350 tablet_service.cc:1467] Processing CreateTablet for tablet bdb19a28f07c4e3d80061d63a5817a81 (DEFAULT_TABLE table=test-table [id=f684b7dc1de94a36a7652b3b694101f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:48.419764 17350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb19a28f07c4e3d80061d63a5817a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:48.419764 17495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb19a28f07c4e3d80061d63a5817a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:48.421862 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.412808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.28.108.129:40435)
0807 08:46:48.413013 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:48.413018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:48.413045 (+ 27us) server_negotiation.cc:407] Connection header received
0807 08:46:48.413090 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:48.413092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:48.413109 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:48.413534 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:48.413824 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.415097 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.416595 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.417380 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.417498 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.417606 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:48.421700 (+ 4094us) server_negotiation.cc:299] Negotiation successful
0807 08:46:48.421777 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":109,"threads_started":1}
I20260807 08:46:48.424038 17583 tablet_bootstrap.cc:492] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07: Bootstrap starting.
I20260807 08:46:48.424762 17209 tablet_service.cc:1467] Processing CreateTablet for tablet bdb19a28f07c4e3d80061d63a5817a81 (DEFAULT_TABLE table=test-table [id=f684b7dc1de94a36a7652b3b694101f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:48.425109 17583 tablet_bootstrap.cc:654] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:48.425313 17209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb19a28f07c4e3d80061d63a5817a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:48.425686 17583 log.cc:824] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:48.426191 17582 tablet_bootstrap.cc:492] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: Bootstrap starting.
I20260807 08:46:48.426908 17582 tablet_bootstrap.cc:654] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:48.427189 17582 log.cc:824] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:48.427862 17583 tablet_bootstrap.cc:492] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07: No bootstrap required, opened a new log
I20260807 08:46:48.427971 17583 ts_tablet_manager.cc:1397] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260807 08:46:48.428128 17582 tablet_bootstrap.cc:492] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: No bootstrap required, opened a new log
I20260807 08:46:48.428236 17582 ts_tablet_manager.cc:1397] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 08:46:48.429664 17587 tablet_bootstrap.cc:492] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66: Bootstrap starting.
I20260807 08:46:48.430583 17587 tablet_bootstrap.cc:654] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:48.430845 17582 raft_consensus.cc:348] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3096bb809334cf0a25ce299d04a99c4" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 39533 } } peers { permanent_uuid: "df5cac93a1294a949962f4418bf51e66" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40435 } } peers { permanent_uuid: "baf95a85111248a08fdcaa397c523c07" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34747 } }
I20260807 08:46:48.431011 17582 raft_consensus.cc:374] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:48.431048 17582 raft_consensus.cc:729] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3096bb809334cf0a25ce299d04a99c4, State: Initialized, Role: FOLLOWER
I20260807 08:46:48.431226 17582 consensus_queue.cc:260] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3096bb809334cf0a25ce299d04a99c4" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 39533 } } peers { permanent_uuid: "df5cac93a1294a949962f4418bf51e66" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40435 } } peers { permanent_uuid: "baf95a85111248a08fdcaa397c523c07" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34747 } }
I20260807 08:46:48.431143 17583 raft_consensus.cc:348] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3096bb809334cf0a25ce299d04a99c4" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 39533 } } peers { permanent_uuid: "df5cac93a1294a949962f4418bf51e66" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40435 } } peers { permanent_uuid: "baf95a85111248a08fdcaa397c523c07" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34747 } }
I20260807 08:46:48.431345 17583 raft_consensus.cc:374] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:48.431399 17583 raft_consensus.cc:729] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf95a85111248a08fdcaa397c523c07, State: Initialized, Role: FOLLOWER
I20260807 08:46:48.431279 17587 log.cc:824] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:48.431558 17582 ts_tablet_manager.cc:1428] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 08:46:48.431555 17583 consensus_queue.cc:260] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3096bb809334cf0a25ce299d04a99c4" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 39533 } } peers { permanent_uuid: "df5cac93a1294a949962f4418bf51e66" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40435 } } peers { permanent_uuid: "baf95a85111248a08fdcaa397c523c07" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34747 } }
I20260807 08:46:48.431768 17425 heartbeater.cc:503] Master 127.28.108.190:33675 was elected leader, sending a full tablet report...
I20260807 08:46:48.431893 17583 ts_tablet_manager.cc:1428] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260807 08:46:48.431957 17566 heartbeater.cc:503] Master 127.28.108.190:33675 was elected leader, sending a full tablet report...
I20260807 08:46:48.432753 17587 tablet_bootstrap.cc:492] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66: No bootstrap required, opened a new log
I20260807 08:46:48.432964 17587 ts_tablet_manager.cc:1397] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 08:46:48.435568 17587 raft_consensus.cc:348] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3096bb809334cf0a25ce299d04a99c4" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 39533 } } peers { permanent_uuid: "df5cac93a1294a949962f4418bf51e66" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40435 } } peers { permanent_uuid: "baf95a85111248a08fdcaa397c523c07" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34747 } }
I20260807 08:46:48.435757 17587 raft_consensus.cc:374] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:48.435784 17587 raft_consensus.cc:729] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5cac93a1294a949962f4418bf51e66, State: Initialized, Role: FOLLOWER
I20260807 08:46:48.435925 17587 consensus_queue.cc:260] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3096bb809334cf0a25ce299d04a99c4" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 39533 } } peers { permanent_uuid: "df5cac93a1294a949962f4418bf51e66" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40435 } } peers { permanent_uuid: "baf95a85111248a08fdcaa397c523c07" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34747 } }
I20260807 08:46:48.436216 17587 ts_tablet_manager.cc:1428] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 08:46:48.436365 17280 heartbeater.cc:503] Master 127.28.108.190:33675 was elected leader, sending a full tablet report...
W20260807 08:46:48.472071 17426 tablet.cc:2369] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:46:48.527027 17588 raft_consensus.cc:482] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:48.527212 17588 raft_consensus.cc:504] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3096bb809334cf0a25ce299d04a99c4" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 39533 } } peers { permanent_uuid: "df5cac93a1294a949962f4418bf51e66" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40435 } } peers { permanent_uuid: "baf95a85111248a08fdcaa397c523c07" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34747 } }
I20260807 08:46:48.527719 17588 leader_election.cc:288] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df5cac93a1294a949962f4418bf51e66 (127.28.108.129:40435), baf95a85111248a08fdcaa397c523c07 (127.28.108.131:34747)
I20260807 08:46:48.531543 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.528225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:35619 (local address 127.28.108.129:40435)
0807 08:46:48.528320 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:48.528322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:48.528337 (+ 15us) server_negotiation.cc:407] Connection header received
0807 08:46:48.528366 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:48.528367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:48.528378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:48.528463 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:48.528669 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.529242 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.530578 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.531130 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.531219 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.531232 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:48.531409 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 08:46:48.531457 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 08:46:48.531543 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.527961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:40435 (local address 127.28.108.130:35619)
0807 08:46:48.528073 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:48.528088 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:48.528151 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:48.528486 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:48.528487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:48.528489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:48.528578 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.528579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.529464 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.529466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.530460 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.530463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.531231 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.531232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.531333 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.531339 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 08:46:48.531400 (+ 61us) client_negotiation.cc:240] Negotiation successful
0807 08:46:48.531453 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":2}
I20260807 08:46:48.531648 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.528130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:56325 (local address 127.28.108.131:34747)
0807 08:46:48.528179 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:48.528181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:48.528455 (+ 274us) server_negotiation.cc:407] Connection header received
0807 08:46:48.528564 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:48.528565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:48.528576 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:48.528655 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:48.528867 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.529403 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.530593 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.531225 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.531305 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.531316 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:48.531514 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 08:46:48.531558 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 08:46:48.531926 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.528038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:34747 (local address 127.28.108.130:56325)
0807 08:46:48.528269 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:48.528281 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:48.528475 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:48.528674 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:48.528675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:48.528677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:48.528774 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.528777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.529507 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.529508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.530481 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:48.530483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.531309 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.531310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:48.531420 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.531423 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:46:48.531504 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 08:46:48.531551 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":5,"thread_start_us":154,"threads_started":1}
I20260807 08:46:48.531944 17230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb19a28f07c4e3d80061d63a5817a81" candidate_uuid: "f3096bb809334cf0a25ce299d04a99c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5cac93a1294a949962f4418bf51e66" is_pre_election: true
I20260807 08:46:48.532135 17230 raft_consensus.cc:2454] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3096bb809334cf0a25ce299d04a99c4 in term 0.
I20260807 08:46:48.532363 17515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb19a28f07c4e3d80061d63a5817a81" candidate_uuid: "f3096bb809334cf0a25ce299d04a99c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf95a85111248a08fdcaa397c523c07" is_pre_election: true
I20260807 08:46:48.532532 17515 raft_consensus.cc:2454] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3096bb809334cf0a25ce299d04a99c4 in term 0.
I20260807 08:46:48.532512 17301 leader_election.cc:302] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df5cac93a1294a949962f4418bf51e66, f3096bb809334cf0a25ce299d04a99c4; no voters:
I20260807 08:46:48.532701 17588 raft_consensus.cc:2781] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:46:48.532792 17588 raft_consensus.cc:482] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:46:48.532847 17588 raft_consensus.cc:3037] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:48.533730 17588 raft_consensus.cc:504] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3096bb809334cf0a25ce299d04a99c4" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 39533 } } peers { permanent_uuid: "df5cac93a1294a949962f4418bf51e66" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40435 } } peers { permanent_uuid: "baf95a85111248a08fdcaa397c523c07" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34747 } }
I20260807 08:46:48.533927 17588 leader_election.cc:288] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [CANDIDATE]: Term 1 election: Requested vote from peers df5cac93a1294a949962f4418bf51e66 (127.28.108.129:40435), baf95a85111248a08fdcaa397c523c07 (127.28.108.131:34747)
I20260807 08:46:48.534114 17515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb19a28f07c4e3d80061d63a5817a81" candidate_uuid: "f3096bb809334cf0a25ce299d04a99c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf95a85111248a08fdcaa397c523c07"
I20260807 08:46:48.534222 17515 raft_consensus.cc:3037] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:48.534317 17230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb19a28f07c4e3d80061d63a5817a81" candidate_uuid: "f3096bb809334cf0a25ce299d04a99c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5cac93a1294a949962f4418bf51e66"
I20260807 08:46:48.534440 17230 raft_consensus.cc:3037] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:48.535219 17515 raft_consensus.cc:2454] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3096bb809334cf0a25ce299d04a99c4 in term 1.
I20260807 08:46:48.535450 17303 leader_election.cc:302] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baf95a85111248a08fdcaa397c523c07, f3096bb809334cf0a25ce299d04a99c4; no voters:
I20260807 08:46:48.535473 17230 raft_consensus.cc:2454] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3096bb809334cf0a25ce299d04a99c4 in term 1.
I20260807 08:46:48.535724 17588 raft_consensus.cc:2781] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:48.535912 17588 raft_consensus.cc:686] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 1 LEADER]: Becoming Leader. State: Replica: f3096bb809334cf0a25ce299d04a99c4, State: Running, Role: LEADER
I20260807 08:46:48.536052 17588 consensus_queue.cc:237] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3096bb809334cf0a25ce299d04a99c4" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 39533 } } peers { permanent_uuid: "df5cac93a1294a949962f4418bf51e66" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40435 } } peers { permanent_uuid: "baf95a85111248a08fdcaa397c523c07" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34747 } }
I20260807 08:46:48.538159 17059 catalog_manager.cc:5495] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3096bb809334cf0a25ce299d04a99c4 (127.28.108.130). New cstate: current_term: 1 leader_uuid: "f3096bb809334cf0a25ce299d04a99c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3096bb809334cf0a25ce299d04a99c4" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 39533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df5cac93a1294a949962f4418bf51e66" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baf95a85111248a08fdcaa397c523c07" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34747 } health_report { overall_health: UNKNOWN } } }
I20260807 08:46:48.566713 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.563662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.28.108.130:39533)
0807 08:46:48.563782 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:48.563788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:48.563798 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:46:48.563843 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:48.563844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:48.563873 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:48.563944 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:46:48.564085 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.564573 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.565597 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:48.565878 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:48.565960 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:48.566152 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:46:48.566267 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:46:48.566351 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:48.566544 (+ 193us) server_negotiation.cc:299] Negotiation successful
0807 08:46:48.566563 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 08:46:48.569175 17515 raft_consensus.cc:1261] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07 [term 1 FOLLOWER]: Refusing update from remote peer f3096bb809334cf0a25ce299d04a99c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:48.569175 17230 raft_consensus.cc:1261] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66 [term 1 FOLLOWER]: Refusing update from remote peer f3096bb809334cf0a25ce299d04a99c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:48.569795 17593 consensus_queue.cc:1035] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf95a85111248a08fdcaa397c523c07" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:48.570113 17588 consensus_queue.cc:1035] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5cac93a1294a949962f4418bf51e66" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 08:46:48.572817 17281 tablet.cc:2369] T bdb19a28f07c4e3d80061d63a5817a81 P df5cac93a1294a949962f4418bf51e66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:46:48.573819 17599 mvcc.cc:205] Tried to move back new op lower bound from 7315834505495539712 to 7315834505365712896. Current Snapshot: MvccSnapshot[applied={T|T < 7315834505495539712}]
I20260807 08:46:48.573977 17597 mvcc.cc:205] Tried to move back new op lower bound from 7315834505495539712 to 7315834505365712896. Current Snapshot: MvccSnapshot[applied={T|T < 7315834505495539712}]
I20260807 08:46:48.576617 17059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:48.576750 17059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:48.577849 17059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e0db6836ced46f78883513df044f13f: Sending DeleteTablet for 3 replicas of tablet bdb19a28f07c4e3d80061d63a5817a81
I20260807 08:46:48.578259 17350 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb19a28f07c4e3d80061d63a5817a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:48 UTC) from {username='slave'} at 127.0.0.1:35736
I20260807 08:46:48.578349 17209 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb19a28f07c4e3d80061d63a5817a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:48 UTC) from {username='slave'} at 127.0.0.1:34808
I20260807 08:46:48.578538 17495 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb19a28f07c4e3d80061d63a5817a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:48 UTC) from {username='slave'} at 127.0.0.1:44590
I20260807 08:46:48.578616 17608 tablet_replica.cc:330] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: stopping tablet replica
I20260807 08:46:48.578725 17608 raft_consensus.cc:2229] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:46:48.578882 17608 raft_consensus.cc:2258] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:48.579034 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 17137
I20260807 08:46:48.579444 17608 ts_tablet_manager.cc:1905] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:48.581425 17610 tablet_replica.cc:330] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07: stopping tablet replica
I20260807 08:46:48.581729 17608 ts_tablet_manager.cc:1918] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:48.581833 17608 log.cc:1196] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/bdb19a28f07c4e3d80061d63a5817a81
I20260807 08:46:48.582129 17608 ts_tablet_manager.cc:1939] T bdb19a28f07c4e3d80061d63a5817a81 P f3096bb809334cf0a25ce299d04a99c4: Deleting consensus metadata
I20260807 08:46:48.581590 17610 raft_consensus.cc:2229] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:48.582473 17610 raft_consensus.cc:2258] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:48.583057 17045 catalog_manager.cc:4841] TS f3096bb809334cf0a25ce299d04a99c4 (127.28.108.130:39533): tablet bdb19a28f07c4e3d80061d63a5817a81 (table test-table [id=f684b7dc1de94a36a7652b3b694101f7]) successfully deleted
I20260807 08:46:48.583606 17610 ts_tablet_manager.cc:1905] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:48.586071 17610 ts_tablet_manager.cc:1918] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:48.586191 17610 log.cc:1196] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/bdb19a28f07c4e3d80061d63a5817a81
I20260807 08:46:48.586520 17610 ts_tablet_manager.cc:1939] T bdb19a28f07c4e3d80061d63a5817a81 P baf95a85111248a08fdcaa397c523c07: Deleting consensus metadata
I20260807 08:46:48.587524 17047 catalog_manager.cc:4841] TS baf95a85111248a08fdcaa397c523c07 (127.28.108.131:34747): tablet bdb19a28f07c4e3d80061d63a5817a81 (table test-table [id=f684b7dc1de94a36a7652b3b694101f7]) successfully deleted
W20260807 08:46:48.592536 17045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:40435 (error 108)
I20260807 08:46:48.592729 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 17283
I20260807 08:46:48.596388 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:48.596206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:40435 (local address 127.0.0.1:34814)
0807 08:46:48.596279 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:48.596340 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:40435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 08:46:48.596637 17045 catalog_manager.cc:4584] TS df5cac93a1294a949962f4418bf51e66 (127.28.108.129:40435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdb19a28f07c4e3d80061d63a5817a81: Network error: Client connection negotiation failed: client connection to 127.28.108.129:40435: connect: Connection refused (error 111)
I20260807 08:46:48.600535 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 17428
I20260807 08:46:48.608358 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 17025
2026-08-07T08:46:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (912 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:46:48 dist-test-slave-9zdj krb5kdc[17614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:46:48 dist-test-slave-9zdj krb5kdc[17614](info): set up 2 sockets
Aug 07 08:46:48 dist-test-slave-9zdj krb5kdc[17614](info): commencing operation
krb5kdc: starting...
W20260807 08:46:49.736603 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:46:49 dist-test-slave-9zdj krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:46:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:46:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:46:49.890146 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:35733
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:39941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:35733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:49.982061 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:49.982388 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:49.982438 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:49.983757 17627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:49.983842 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:49.983866 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:49.983888 17627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:49.983915 17627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:49.985863 17627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:35733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:35733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:49.986196 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:49.986433 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:49.992264 17634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:49.992441 17627 server_base.cc:1034] running on GCE node
W20260807 08:46:49.992252 17636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:49.992252 17633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:49.992946 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:49.993381 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:49.994565 17627 hybrid_clock.cc:648] HybridClock initialized: now 1786092409994530 us; error 50 us; skew 500 ppm
Aug 07 08:46:50 dist-test-slave-9zdj krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092410, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:50.002756 17627 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:46:50.003456 17627 webserver.cc:460] Webserver started at http://127.28.108.190:39697/ using document root <none> and password file <none>
I20260807 08:46:50.003752 17627 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:50.003835 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:50.004010 17627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:50.005291 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "ba07ba90a92c4fa9904330a5874d125f"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "1915d1049519d6f6151413b4286f18ec"
server_key_iv: "40c24d89e4334125b6e2b9b4fc4a9352"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.005793 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "ba07ba90a92c4fa9904330a5874d125f"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "1915d1049519d6f6151413b4286f18ec"
server_key_iv: "40c24d89e4334125b6e2b9b4fc4a9352"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.007628 17627 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260807 08:46:50.008668 17643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:50.009032 17627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:46:50.009195 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "ba07ba90a92c4fa9904330a5874d125f"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "1915d1049519d6f6151413b4286f18ec"
server_key_iv: "40c24d89e4334125b6e2b9b4fc4a9352"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.009294 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:50.030480 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:50.033901 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:50.034060 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:50.039074 17627 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:35733
I20260807 08:46:50.039165 17702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:35733 every 8 connection(s)
I20260807 08:46:50.039600 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:50.040725 17703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:50.043630 17703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f: Bootstrap starting.
I20260807 08:46:50.044373 17703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:50.044734 17703 log.cc:824] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:50.045872 17703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f: No bootstrap required, opened a new log
I20260807 08:46:50.047748 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 17627
I20260807 08:46:50.047853 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:46:50.047837 17703 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba07ba90a92c4fa9904330a5874d125f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35733 } }
I20260807 08:46:50.048081 17703 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:50.048089 29106 external_mini_cluster.cc:1428] Setting key 333ffb2ebf33fcdc3f3e399e024532c6
I20260807 08:46:50.048137 17703 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba07ba90a92c4fa9904330a5874d125f, State: Initialized, Role: FOLLOWER
I20260807 08:46:50.048341 17703 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba07ba90a92c4fa9904330a5874d125f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35733 } }
I20260807 08:46:50.048452 17703 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:50.048496 17703 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:50.048547 17703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:50.049350 17703 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba07ba90a92c4fa9904330a5874d125f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35733 } }
I20260807 08:46:50.049491 17703 leader_election.cc:302] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba07ba90a92c4fa9904330a5874d125f; no voters:
I20260807 08:46:50.049746 17703 leader_election.cc:288] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:46:50.050009 17709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:50.050069 17703 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:50.050184 17709 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [term 1 LEADER]: Becoming Leader. State: Replica: ba07ba90a92c4fa9904330a5874d125f, State: Running, Role: LEADER
I20260807 08:46:50.050331 17709 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba07ba90a92c4fa9904330a5874d125f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35733 } }
I20260807 08:46:50.050733 17709 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba07ba90a92c4fa9904330a5874d125f. Latest consensus state: current_term: 1 leader_uuid: "ba07ba90a92c4fa9904330a5874d125f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba07ba90a92c4fa9904330a5874d125f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35733 } } }
I20260807 08:46:50.050782 17711 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba07ba90a92c4fa9904330a5874d125f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba07ba90a92c4fa9904330a5874d125f" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 35733 } } }
I20260807 08:46:50.050884 17709 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:50.051005 17711 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:50.051242 17718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:50.052090 17718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:50.054003 17718 catalog_manager.cc:1347] Generated new cluster ID: 11ddfcf942f941fea2029ec090848913
I20260807 08:46:50.054066 17718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:46:50 dist-test-slave-9zdj krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:50.061828 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.048708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36998 (local address 127.28.108.190:35733)
0807 08:46:50.048919 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:50.048930 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:50.048968 (+ 38us) server_negotiation.cc:407] Connection header received
0807 08:46:50.049446 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:50.049453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:50.049546 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:50.049762 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:50.050689 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.052420 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.053583 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.054324 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.054467 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.057236 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:50.057242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:50.057242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:50.057257 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:50.059882 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:50.060334 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:50.060338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:50.060338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:50.060475 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:50.060736 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:50.060737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:50.060737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:50.061064 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:50.061279 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:50.061372 (+ 93us) server_negotiation.cc:299] Negotiation successful
0807 08:46:50.061542 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":120,"threads_started":1}
I20260807 08:46:50.064996 17718 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:46:50.065964 17718 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:50.074414 17718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f: Generated new TSK 0
I20260807 08:46:50.074677 17718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.108.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
Entry for principal kudu/127.28.108.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab.
I20260807 08:46:50.123450 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35733
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:39941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:50.241519 17735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:50.241820 17735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:50.241854 17735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:50.243551 17735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:50.243640 17735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:50.243677 17735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:46:50.245811 17735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.129.keytab
--principal=kudu/127.28.108.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:50.246171 17735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:50.246486 17735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:50.253648 17741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:50.253655 17740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:50.253648 17743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:50.253674 17735 server_base.cc:1034] running on GCE node
I20260807 08:46:50.254330 17735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:50.254676 17735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:50.255846 17735 hybrid_clock.cc:648] HybridClock initialized: now 1786092410255863 us; error 69 us; skew 500 ppm
Aug 07 08:46:50 dist-test-slave-9zdj krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092410, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:50.259299 17735 init.cc:377] Logged in from keytab as kudu/127.28.108.129@KRBTEST.COM (short username kudu)
I20260807 08:46:50.259931 17735 webserver.cc:460] Webserver started at http://127.28.108.129:33237/ using document root <none> and password file <none>
I20260807 08:46:50.260221 17735 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:50.260308 17735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:50.260490 17735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:50.261883 17735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "b93082922247449998e0a34ced0dd511"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "310ab8f61ef62c387dbed42ae8e3be3e"
server_key_iv: "a41896cbda00e0bc619ababdb6ceb755"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.262297 17735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "b93082922247449998e0a34ced0dd511"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "310ab8f61ef62c387dbed42ae8e3be3e"
server_key_iv: "a41896cbda00e0bc619ababdb6ceb755"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.263917 17735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:46:50.264914 17753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:50.265233 17735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:50.265373 17735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "b93082922247449998e0a34ced0dd511"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "310ab8f61ef62c387dbed42ae8e3be3e"
server_key_iv: "a41896cbda00e0bc619ababdb6ceb755"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.265472 17735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:50.283304 17735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:50.286562 17735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:50.286808 17735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:50.287180 17735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:50.287599 17735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:50.287670 17735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:50.287719 17735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:50.287752 17735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:50.295378 17735 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:40611
I20260807 08:46:50.295434 17874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:40611 every 8 connection(s)
I20260807 08:46:50.295835 17735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:46:50.302583 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 17735
I20260807 08:46:50.302853 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:46:50.303179 29106 external_mini_cluster.cc:1428] Setting key 1b2092dc34dc06125794fe00c2c99414
Aug 07 08:46:50 dist-test-slave-9zdj krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092410, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.129@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:50.307721 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.296758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:48261 (local address 127.28.108.190:35733)
0807 08:46:50.296845 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:50.296848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:50.297049 (+ 201us) server_negotiation.cc:407] Connection header received
0807 08:46:50.297766 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:50.297768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:50.297786 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:50.297857 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:50.298962 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.299532 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.300669 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.300888 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.300962 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.304130 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:50.304132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:50.304133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:50.304142 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:50.306036 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:50.306597 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:50.306599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:50.306600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:50.306670 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:50.306968 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:50.306969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:50.306969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:50.307101 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:50.307189 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:50.307507 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 08:46:50.307620 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 08:46:50.307951 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.296704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35733 (local address 127.28.108.129:48261)
0807 08:46:50.296896 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:50.296940 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:50.297571 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:50.297935 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:50.297939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:50.298591 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:50.298821 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.298826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.299654 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.299655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.300555 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.300558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.301018 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.301020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.301174 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.302075 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:50.302086 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:50.304020 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:50.306179 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:50.306192 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:50.306195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:50.306500 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:50.306747 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:50.306748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:50.306749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:50.306887 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:50.307180 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:50.307190 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:50.307360 (+ 170us) client_negotiation.cc:769] Sending connection context
0807 08:46:50.307483 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 08:46:50.307683 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":111,"threads_started":1}
I20260807 08:46:50.308545 17875 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35733
I20260807 08:46:50.308682 17875 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:50.308918 17875 heartbeater.cc:511] Master 127.28.108.190:35733 requested a full tablet report, sending...
I20260807 08:46:50.309825 17663 ts_manager.cc:194] Registered new tserver with Master: b93082922247449998e0a34ced0dd511 (127.28.108.129:40611)
I20260807 08:46:50.310612 17663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.129@KRBTEST.COM'} at 127.28.108.129:48261
WARNING: no policy specified for kudu/127.28.108.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
Entry for principal kudu/127.28.108.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab.
I20260807 08:46:50.337180 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35733
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:39941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:50.426633 17880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:50.426956 17880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:50.427008 17880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:50.428412 17880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:50.428520 17880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:50.428571 17880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:46:50.430399 17880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.130.keytab
--principal=kudu/127.28.108.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:50.430763 17880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:50.431041 17880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:50.437271 17887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:50.437357 17885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:50.437399 17889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:50.437417 17880 server_base.cc:1034] running on GCE node
I20260807 08:46:50.437879 17880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:50.438151 17880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:50.439353 17880 hybrid_clock.cc:648] HybridClock initialized: now 1786092410439327 us; error 48 us; skew 500 ppm
Aug 07 08:46:50 dist-test-slave-9zdj krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092410, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:50.442812 17880 init.cc:377] Logged in from keytab as kudu/127.28.108.130@KRBTEST.COM (short username kudu)
I20260807 08:46:50.443382 17880 webserver.cc:460] Webserver started at http://127.28.108.130:46055/ using document root <none> and password file <none>
I20260807 08:46:50.443643 17880 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:50.443696 17880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:50.443961 17880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:50.445276 17880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "4cb722e3cebe4ad198e687f165c9dfc5"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "ee7a9e061b8e11f9884511fe7415039c"
server_key_iv: "812de3a3a5f688a6a8d8e746ba92d4b1"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.445713 17880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "4cb722e3cebe4ad198e687f165c9dfc5"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "ee7a9e061b8e11f9884511fe7415039c"
server_key_iv: "812de3a3a5f688a6a8d8e746ba92d4b1"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.447256 17880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:50.448395 17896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:50.448993 17880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:50.449096 17880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "4cb722e3cebe4ad198e687f165c9dfc5"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "ee7a9e061b8e11f9884511fe7415039c"
server_key_iv: "812de3a3a5f688a6a8d8e746ba92d4b1"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.449311 17880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:50.461930 17880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:50.464920 17880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:50.465058 17880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:50.465377 17880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:50.465853 17880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:50.465915 17880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:50.465951 17880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:50.465979 17880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:50.472540 17880 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:37083
I20260807 08:46:50.472600 18033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:37083 every 8 connection(s)
I20260807 08:46:50.473021 17880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:46:50.474746 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 17880
I20260807 08:46:50.474858 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:46:50.475106 29106 external_mini_cluster.cc:1428] Setting key c450b42c31a43bd3a26f3bd45e3f29b6
Aug 07 08:46:50 dist-test-slave-9zdj krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092410, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.130@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.108.131@KRBTEST.COM; defaulting to no policy
I20260807 08:46:50.485797 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.474010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:45731 (local address 127.28.108.190:35733)
0807 08:46:50.474098 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:50.474101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:50.474464 (+ 363us) server_negotiation.cc:407] Connection header received
0807 08:46:50.475173 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:50.475175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:50.475190 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:50.475234 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:50.476481 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.477005 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.478453 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.478721 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.478812 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.482057 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:50.482060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:50.482060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:50.482076 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:50.483887 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:50.484449 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:50.484451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:50.484451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:50.484524 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:50.484852 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:50.484853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:50.484854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:50.485004 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:50.485089 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:50.485548 (+ 459us) server_negotiation.cc:299] Negotiation successful
0807 08:46:50.485702 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 08:46:50.485934 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.473991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35733 (local address 127.28.108.130:45731)
0807 08:46:50.474294 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:50.474351 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:50.475044 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:50.475338 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:50.475343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:50.476121 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:50.476356 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.476362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.477122 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.477123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.478220 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.478224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.478864 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.478865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.479058 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.479809 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:50.479823 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:50.481905 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:50.484006 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:50.484019 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:50.484023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:50.484339 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:50.484614 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:50.484615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:50.484615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:50.484770 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:50.485092 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:50.485101 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:50.485297 (+ 196us) client_negotiation.cc:769] Sending connection context
0807 08:46:50.485449 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 08:46:50.485688 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260807 08:46:50.487041 18034 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35733
I20260807 08:46:50.487174 18034 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:50.487495 18034 heartbeater.cc:511] Master 127.28.108.190:35733 requested a full tablet report, sending...
I20260807 08:46:50.488129 17663 ts_manager.cc:194] Registered new tserver with Master: 4cb722e3cebe4ad198e687f165c9dfc5 (127.28.108.130:37083)
I20260807 08:46:50.488715 17663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.130@KRBTEST.COM'} at 127.28.108.130:45731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
Entry for principal kudu/127.28.108.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab.
I20260807 08:46:50.510960 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35733
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:39941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:50.602211 18039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:50.602463 18039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:50.602489 18039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:50.603706 18039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:50.603750 18039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:50.603780 18039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:46:50.605469 18039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:39941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.131.keytab
--principal=kudu/127.28.108.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:35733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:50.605779 18039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:50.606040 18039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:50.611974 18050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:50.611995 18048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:50.611984 18045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:50.612439 18039 server_base.cc:1034] running on GCE node
I20260807 08:46:50.612648 18039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:50.612944 18039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:50.614141 18039 hybrid_clock.cc:648] HybridClock initialized: now 1786092410614065 us; error 100 us; skew 500 ppm
Aug 07 08:46:50 dist-test-slave-9zdj krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092410, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:50.617712 18039 init.cc:377] Logged in from keytab as kudu/127.28.108.131@KRBTEST.COM (short username kudu)
I20260807 08:46:50.618347 18039 webserver.cc:460] Webserver started at http://127.28.108.131:44047/ using document root <none> and password file <none>
I20260807 08:46:50.618629 18039 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:50.618705 18039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:50.618909 18039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:50.620093 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "9e8e209c1540cc13c2432c1c5593136c"
server_key_iv: "95d26ed0ec47b2f9fd930886f895364b"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.620501 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "9e8e209c1540cc13c2432c1c5593136c"
server_key_iv: "95d26ed0ec47b2f9fd930886f895364b"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.622227 18039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 08:46:50.623503 18058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:50.623798 18039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:50.623977 18039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2"
format_stamp: "Formatted at 2026-08-07 08:46:50 on dist-test-slave-9zdj"
server_key: "9e8e209c1540cc13c2432c1c5593136c"
server_key_iv: "95d26ed0ec47b2f9fd930886f895364b"
server_key_version: "encryptionkey@0"
I20260807 08:46:50.624073 18039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:50.653173 18039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:50.655990 18039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:50.656196 18039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:50.656502 18039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:50.656883 18039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:50.656947 18039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:50.657025 18039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:50.657060 18039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:50.664783 18039 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:40243
I20260807 08:46:50.664978 18185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:40243 every 8 connection(s)
I20260807 08:46:50.665347 18039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:46:50.668823 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 18039
I20260807 08:46:50.669003 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:46:50.669234 29106 external_mini_cluster.cc:1428] Setting key b4a40ab63f6ae639e86906367fb93946
Aug 07 08:46:50 dist-test-slave-9zdj krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092410, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.131@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:46:50.678381 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.666467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:42283 (local address 127.28.108.190:35733)
0807 08:46:50.666534 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:50.666536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:50.667367 (+ 831us) server_negotiation.cc:407] Connection header received
0807 08:46:50.667907 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:50.667909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:50.667920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:50.667997 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:50.669580 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.670419 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.671641 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.671878 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.671948 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.674747 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:50.674749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:50.674750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:50.674768 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:50.676384 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:50.676952 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:50.676954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:50.676954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:50.677032 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:50.677360 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:50.677362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:50.677362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:50.677566 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:50.677672 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:50.678147 (+ 475us) server_negotiation.cc:299] Negotiation successful
0807 08:46:50.678275 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260807 08:46:50.678640 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.666415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:35733 (local address 127.28.108.131:42283)
0807 08:46:50.667216 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:50.667264 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:50.667801 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:50.668060 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:50.668063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:50.669030 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:50.669417 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.669426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.670536 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.670536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.671521 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.671524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.671978 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.671979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.672109 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.672838 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:46:50.672848 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:50.674604 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:50.676539 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:50.676549 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:50.676552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:50.676852 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:50.677122 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:46:50.677123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:46:50.677124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:46:50.677265 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:46:50.677676 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:50.677686 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:50.677968 (+ 282us) client_negotiation.cc:769] Sending connection context
0807 08:46:50.678118 (+ 150us) client_negotiation.cc:240] Negotiation successful
0807 08:46:50.678324 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":109,"threads_started":1}
I20260807 08:46:50.679282 18186 heartbeater.cc:347] Connected to a master server at 127.28.108.190:35733
I20260807 08:46:50.679399 18186 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:50.679647 18186 heartbeater.cc:511] Master 127.28.108.190:35733 requested a full tablet report, sending...
I20260807 08:46:50.680282 17663 ts_manager.cc:194] Registered new tserver with Master: ab7c7a2ad3b049ef82ca16f07bc3b6c2 (127.28.108.131:40243)
I20260807 08:46:50.680866 17663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.108.131@KRBTEST.COM'} at 127.28.108.131:42283
I20260807 08:46:50.681895 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:46:50.690483 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.683377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.28.108.190:35733)
0807 08:46:50.683423 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:50.683425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:50.683544 (+ 119us) server_negotiation.cc:407] Connection header received
0807 08:46:50.683585 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:50.683586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:50.683595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:50.683655 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:50.684239 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.684824 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.685722 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.685936 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.686001 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.687326 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:50.687327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:50.687327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:50.687335 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:50.688971 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:50.689381 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:50.689383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:50.689383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:50.689449 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:50.689753 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:50.689754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:50.689755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:50.689939 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:50.690029 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:50.690267 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 08:46:50.690349 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260807 08:46:50.692444 17663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:46:50.694406 17663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:46:50.702361 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.697340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:40611 (local address 127.0.0.1:34746)
0807 08:46:50.697531 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:50.697576 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:50.697794 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:50.698166 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:50.698170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:50.698173 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:50.698260 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.698262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.699867 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.699868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.700873 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.700876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.701999 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.702000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.702155 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.702160 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:46:50.702211 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 08:46:50.702254 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260807 08:46:50.702538 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.697340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:40243 (local address 127.0.0.1:60866)
0807 08:46:50.697535 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:50.697576 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:50.697675 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:50.698466 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:50.698466 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:50.698468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:50.698515 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.698517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.700079 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.700079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.701120 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.701122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.702240 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.702241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.702328 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.702333 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:46:50.702392 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 08:46:50.702441 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260807 08:46:50.702785 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.697393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:37083 (local address 127.0.0.1:39876)
0807 08:46:50.697695 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:50.697704 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:50.697743 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:50.698562 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:50.698563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:50.698564 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:50.698604 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.698605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.699488 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.699489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.700914 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.700917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.701789 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.701790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.701893 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.701910 (+ 17us) client_negotiation.cc:769] Sending connection context
0807 08:46:50.702009 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 08:46:50.702077 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":46,"thread_start_us":110,"threads_started":1}
I20260807 08:46:50.702816 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.697543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.28.108.131:40243)
0807 08:46:50.698249 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:50.698254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:50.698265 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:46:50.698337 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:50.698339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:50.698358 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:50.698456 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:50.698581 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.699938 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.701219 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.702122 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.702239 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.702299 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:50.702675 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 08:46:50.702726 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":133,"threads_started":1}
I20260807 08:46:50.702893 18202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.697987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.28.108.130:37083)
0807 08:46:50.698183 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:50.698188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:50.698198 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:46:50.698236 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:50.698239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:50.698254 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:50.698468 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:50.698664 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.699373 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.701033 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.701694 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.701792 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.701860 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:50.702412 (+ 552us) server_negotiation.cc:299] Negotiation successful
0807 08:46:50.702462 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
I20260807 08:46:50.702838 18201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.697737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34746 (local address 127.28.108.129:40611)
0807 08:46:50.697940 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:50.697946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:50.697960 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:46:50.698013 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:50.698015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:50.698035 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:50.698139 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:50.698359 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.699705 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.701000 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.701901 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.701994 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.702057 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:50.702698 (+ 641us) server_negotiation.cc:299] Negotiation successful
0807 08:46:50.702752 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
I20260807 08:46:50.703727 18111 tablet_service.cc:1467] Processing CreateTablet for tablet 6f16b96bfc3f49b9969e728f86201337 (DEFAULT_TABLE table=test-table [id=2bbc3a69c44b4f73b56838fbc675540a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:50.703727 17951 tablet_service.cc:1467] Processing CreateTablet for tablet 6f16b96bfc3f49b9969e728f86201337 (DEFAULT_TABLE table=test-table [id=2bbc3a69c44b4f73b56838fbc675540a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:50.703776 17803 tablet_service.cc:1467] Processing CreateTablet for tablet 6f16b96bfc3f49b9969e728f86201337 (DEFAULT_TABLE table=test-table [id=2bbc3a69c44b4f73b56838fbc675540a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:50.704146 18111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f16b96bfc3f49b9969e728f86201337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:50.704146 17951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f16b96bfc3f49b9969e728f86201337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:50.704173 17803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f16b96bfc3f49b9969e728f86201337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:50.707563 18204 tablet_bootstrap.cc:492] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511: Bootstrap starting.
I20260807 08:46:50.707960 18205 tablet_bootstrap.cc:492] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: Bootstrap starting.
I20260807 08:46:50.708632 18205 tablet_bootstrap.cc:654] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:50.708617 18204 tablet_bootstrap.cc:654] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:50.709259 18205 log.cc:824] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:50.709415 18204 log.cc:824] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:50.709700 18206 tablet_bootstrap.cc:492] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: Bootstrap starting.
I20260807 08:46:50.710320 18205 tablet_bootstrap.cc:492] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: No bootstrap required, opened a new log
I20260807 08:46:50.710376 18204 tablet_bootstrap.cc:492] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511: No bootstrap required, opened a new log
I20260807 08:46:50.710393 18206 tablet_bootstrap.cc:654] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:50.710414 18205 ts_tablet_manager.cc:1397] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:46:50.710438 18204 ts_tablet_manager.cc:1397] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:46:50.710880 18206 log.cc:824] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:50.711671 18206 tablet_bootstrap.cc:492] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: No bootstrap required, opened a new log
I20260807 08:46:50.711761 18206 ts_tablet_manager.cc:1397] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:46:50.712821 18205 raft_consensus.cc:348] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 37083 } } peers { permanent_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40243 } } peers { permanent_uuid: "b93082922247449998e0a34ced0dd511" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40611 } }
I20260807 08:46:50.713048 18205 raft_consensus.cc:374] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:50.712836 18204 raft_consensus.cc:348] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 37083 } } peers { permanent_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40243 } } peers { permanent_uuid: "b93082922247449998e0a34ced0dd511" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40611 } }
I20260807 08:46:50.713099 18205 raft_consensus.cc:729] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb722e3cebe4ad198e687f165c9dfc5, State: Initialized, Role: FOLLOWER
I20260807 08:46:50.713130 18204 raft_consensus.cc:374] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:50.713169 18204 raft_consensus.cc:729] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93082922247449998e0a34ced0dd511, State: Initialized, Role: FOLLOWER
I20260807 08:46:50.713303 18204 consensus_queue.cc:260] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 37083 } } peers { permanent_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40243 } } peers { permanent_uuid: "b93082922247449998e0a34ced0dd511" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40611 } }
I20260807 08:46:50.713577 18205 consensus_queue.cc:260] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 37083 } } peers { permanent_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40243 } } peers { permanent_uuid: "b93082922247449998e0a34ced0dd511" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40611 } }
I20260807 08:46:50.713696 18204 ts_tablet_manager.cc:1428] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 08:46:50.713600 18206 raft_consensus.cc:348] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 37083 } } peers { permanent_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40243 } } peers { permanent_uuid: "b93082922247449998e0a34ced0dd511" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40611 } }
I20260807 08:46:50.713845 17875 heartbeater.cc:503] Master 127.28.108.190:35733 was elected leader, sending a full tablet report...
I20260807 08:46:50.713846 18206 raft_consensus.cc:374] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:50.713893 18206 raft_consensus.cc:729] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7c7a2ad3b049ef82ca16f07bc3b6c2, State: Initialized, Role: FOLLOWER
I20260807 08:46:50.713973 18205 ts_tablet_manager.cc:1428] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 08:46:50.714016 18206 consensus_queue.cc:260] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 37083 } } peers { permanent_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40243 } } peers { permanent_uuid: "b93082922247449998e0a34ced0dd511" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40611 } }
I20260807 08:46:50.714123 18034 heartbeater.cc:503] Master 127.28.108.190:35733 was elected leader, sending a full tablet report...
I20260807 08:46:50.714334 18206 ts_tablet_manager.cc:1428] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260807 08:46:50.714502 18186 heartbeater.cc:503] Master 127.28.108.190:35733 was elected leader, sending a full tablet report...
W20260807 08:46:50.723773 18035 tablet.cc:2369] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:46:50.796864 17876 tablet.cc:2369] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 08:46:50.916090 18187 tablet.cc:2369] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:46:50.941690 18211 raft_consensus.cc:482] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:50.941856 18211 raft_consensus.cc:504] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 37083 } } peers { permanent_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40243 } } peers { permanent_uuid: "b93082922247449998e0a34ced0dd511" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40611 } }
I20260807 08:46:50.942399 18211 leader_election.cc:288] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab7c7a2ad3b049ef82ca16f07bc3b6c2 (127.28.108.131:40243), b93082922247449998e0a34ced0dd511 (127.28.108.129:40611)
I20260807 08:46:50.946149 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.942698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:49685 (local address 127.28.108.131:40243)
0807 08:46:50.942761 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:50.942763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:50.942779 (+ 16us) server_negotiation.cc:407] Connection header received
0807 08:46:50.942923 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:50.942924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:50.942940 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:50.943028 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:50.943321 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.943964 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.945003 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.945560 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.945693 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.945704 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:50.946001 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 08:46:50.946056 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260807 08:46:50.946197 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.942603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:40243 (local address 127.28.108.130:49685)
0807 08:46:50.942664 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:50.942678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:50.942800 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:50.943099 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:50.943101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:50.943104 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:50.943200 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.943203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.944072 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.944073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.944888 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.944890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.945776 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.945777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.945881 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.945885 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:46:50.945980 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 08:46:50.946064 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260807 08:46:50.946230 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.942607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:40611 (local address 127.28.108.130:45735)
0807 08:46:50.942949 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:50.942961 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:50.943033 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:50.943283 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:50.943284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:50.943286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:50.943377 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.943378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.944192 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.944193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.945076 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:50.945079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.945836 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.945837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:50.945914 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.945919 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:46:50.946044 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 08:46:50.946082 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":67,"thread_start_us":162,"threads_started":1}
I20260807 08:46:50.946357 18201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.942793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:45735 (local address 127.28.108.129:40611)
0807 08:46:50.942841 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:50.942843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:50.943080 (+ 237us) server_negotiation.cc:407] Connection header received
0807 08:46:50.943118 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:50.943119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:50.943131 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:50.943215 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:50.943487 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.944061 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.945230 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.945730 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.945829 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.945847 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:50.946079 (+ 232us) server_negotiation.cc:299] Negotiation successful
0807 08:46:50.946130 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:46:50.946686 18135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f16b96bfc3f49b9969e728f86201337" candidate_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" is_pre_election: true
I20260807 08:46:50.946719 17823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f16b96bfc3f49b9969e728f86201337" candidate_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b93082922247449998e0a34ced0dd511" is_pre_election: true
I20260807 08:46:50.946892 18135 raft_consensus.cc:2454] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cb722e3cebe4ad198e687f165c9dfc5 in term 0.
I20260807 08:46:50.946897 17823 raft_consensus.cc:2454] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cb722e3cebe4ad198e687f165c9dfc5 in term 0.
I20260807 08:46:50.947212 17900 leader_election.cc:302] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cb722e3cebe4ad198e687f165c9dfc5, ab7c7a2ad3b049ef82ca16f07bc3b6c2; no voters:
I20260807 08:46:50.947358 18211 raft_consensus.cc:2781] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:46:50.947422 18211 raft_consensus.cc:482] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:46:50.947477 18211 raft_consensus.cc:3037] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:50.948208 18211 raft_consensus.cc:504] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 37083 } } peers { permanent_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40243 } } peers { permanent_uuid: "b93082922247449998e0a34ced0dd511" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40611 } }
I20260807 08:46:50.948388 18211 leader_election.cc:288] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [CANDIDATE]: Term 1 election: Requested vote from peers ab7c7a2ad3b049ef82ca16f07bc3b6c2 (127.28.108.131:40243), b93082922247449998e0a34ced0dd511 (127.28.108.129:40611)
I20260807 08:46:50.948629 18135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f16b96bfc3f49b9969e728f86201337" candidate_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2"
I20260807 08:46:50.948629 17823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f16b96bfc3f49b9969e728f86201337" candidate_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b93082922247449998e0a34ced0dd511"
I20260807 08:46:50.948733 18135 raft_consensus.cc:3037] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:50.948742 17823 raft_consensus.cc:3037] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:50.949901 17823 raft_consensus.cc:2454] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cb722e3cebe4ad198e687f165c9dfc5 in term 1.
I20260807 08:46:50.949909 18135 raft_consensus.cc:2454] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cb722e3cebe4ad198e687f165c9dfc5 in term 1.
I20260807 08:46:50.950109 17897 leader_election.cc:302] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cb722e3cebe4ad198e687f165c9dfc5, b93082922247449998e0a34ced0dd511; no voters:
I20260807 08:46:50.950240 18211 raft_consensus.cc:2781] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:50.950336 18211 raft_consensus.cc:686] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 1 LEADER]: Becoming Leader. State: Replica: 4cb722e3cebe4ad198e687f165c9dfc5, State: Running, Role: LEADER
I20260807 08:46:50.950408 18211 consensus_queue.cc:237] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 37083 } } peers { permanent_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40243 } } peers { permanent_uuid: "b93082922247449998e0a34ced0dd511" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40611 } }
I20260807 08:46:50.951460 17661 catalog_manager.cc:5495] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cb722e3cebe4ad198e687f165c9dfc5 (127.28.108.130). New cstate: current_term: 1 leader_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cb722e3cebe4ad198e687f165c9dfc5" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 37083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b93082922247449998e0a34ced0dd511" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40611 } health_report { overall_health: UNKNOWN } } }
I20260807 08:46:50.988372 18202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:50.985674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.28.108.130:37083)
0807 08:46:50.985736 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:50.985738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:50.985749 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:46:50.985825 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:50.985826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:50.985849 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:50.985925 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:46:50.986076 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.986590 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.987494 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:50.987842 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:50.987904 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:50.988064 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:46:50.988179 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:46:50.988239 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:50.988284 (+ 45us) server_negotiation.cc:299] Negotiation successful
0807 08:46:50.988305 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 08:46:50.990749 18135 raft_consensus.cc:1261] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2 [term 1 FOLLOWER]: Refusing update from remote peer 4cb722e3cebe4ad198e687f165c9dfc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:50.990751 17823 raft_consensus.cc:1261] T 6f16b96bfc3f49b9969e728f86201337 P b93082922247449998e0a34ced0dd511 [term 1 FOLLOWER]: Refusing update from remote peer 4cb722e3cebe4ad198e687f165c9dfc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:50.991255 18211 consensus_queue.cc:1035] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7c7a2ad3b049ef82ca16f07bc3b6c2" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 40243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:50.991511 18214 consensus_queue.cc:1035] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b93082922247449998e0a34ced0dd511" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 40611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:50.994865 18217 mvcc.cc:205] Tried to move back new op lower bound from 7315834515414011904 to 7315834515253641216. Current Snapshot: MvccSnapshot[applied={T|T < 7315834515414011904 or (T in {7315834515414011904})}]
I20260807 08:46:50.995424 18218 mvcc.cc:205] Tried to move back new op lower bound from 7315834515414011904 to 7315834515253641216. Current Snapshot: MvccSnapshot[applied={T|T < 7315834515414011904 or (T in {7315834515414011904})}]
I20260807 08:46:50.997283 17661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:50.997413 17661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:50.998317 17661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba07ba90a92c4fa9904330a5874d125f: Sending DeleteTablet for 3 replicas of tablet 6f16b96bfc3f49b9969e728f86201337
I20260807 08:46:50.998752 17951 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f16b96bfc3f49b9969e728f86201337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:50 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:39876
I20260807 08:46:50.998739 17803 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f16b96bfc3f49b9969e728f86201337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:50 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:34746
I20260807 08:46:50.999090 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 17735
I20260807 08:46:50.999187 18229 tablet_replica.cc:330] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: stopping tablet replica
I20260807 08:46:50.999326 18229 raft_consensus.cc:2229] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 1 LEADER]: Raft consensus shutting down.
I20260807 08:46:50.999812 18111 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f16b96bfc3f49b9969e728f86201337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:50 UTC) from {username='kudu', principal='kudu/127.28.108.190@KRBTEST.COM'} at 127.0.0.1:60866
I20260807 08:46:51.005503 18229 raft_consensus.cc:2258] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:51.006016 18230 tablet_replica.cc:330] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: stopping tablet replica
I20260807 08:46:51.006587 18230 raft_consensus.cc:2229] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:51.006837 18230 raft_consensus.cc:2258] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:51.006974 18229 ts_tablet_manager.cc:1905] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:51.007521 18230 ts_tablet_manager.cc:1905] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:51.009404 18229 ts_tablet_manager.cc:1918] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:51.009522 18229 log.cc:1196] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/6f16b96bfc3f49b9969e728f86201337
I20260807 08:46:51.010097 18230 ts_tablet_manager.cc:1918] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:51.010162 18229 ts_tablet_manager.cc:1939] T 6f16b96bfc3f49b9969e728f86201337 P 4cb722e3cebe4ad198e687f165c9dfc5: Deleting consensus metadata
I20260807 08:46:51.010511 18230 log.cc:1196] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/6f16b96bfc3f49b9969e728f86201337
I20260807 08:46:51.010732 17646 catalog_manager.cc:4841] TS 4cb722e3cebe4ad198e687f165c9dfc5 (127.28.108.130:37083): tablet 6f16b96bfc3f49b9969e728f86201337 (table test-table [id=2bbc3a69c44b4f73b56838fbc675540a]) successfully deleted
I20260807 08:46:51.011018 18230 ts_tablet_manager.cc:1939] T 6f16b96bfc3f49b9969e728f86201337 P ab7c7a2ad3b049ef82ca16f07bc3b6c2: Deleting consensus metadata
I20260807 08:46:51.011567 17648 catalog_manager.cc:4841] TS ab7c7a2ad3b049ef82ca16f07bc3b6c2 (127.28.108.131:40243): tablet 6f16b96bfc3f49b9969e728f86201337 (table test-table [id=2bbc3a69c44b4f73b56838fbc675540a]) successfully deleted
W20260807 08:46:51.012844 17644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:40611 (error 108)
I20260807 08:46:51.013469 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.013275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:40611 (local address 127.0.0.1:34750)
0807 08:46:51.013380 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:51.013435 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:40611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 08:46:51.013671 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 17880
W20260807 08:46:51.013680 17644 catalog_manager.cc:4584] TS b93082922247449998e0a34ced0dd511 (127.28.108.129:40611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f16b96bfc3f49b9969e728f86201337: Network error: Client connection negotiation failed: client connection to 127.28.108.129:40611: connect: Connection refused (error 111)
I20260807 08:46:51.023463 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 18039
I20260807 08:46:51.032150 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 17627
2026-08-07T08:46:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2425 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T08:46:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:46:51Z Disabled control of system clock
I20260807 08:46:51.069458 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42743
--webserver_interface=127.28.108.190
--webserver_port=0
--builtin_ntp_servers=127.28.108.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:42743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 08:46:51.164484 18235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:51.164813 18235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:51.164857 18235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:51.166177 18235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:51.166278 18235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:51.166306 18235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:51.166337 18235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:51.166369 18235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:51.167941 18235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:42743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:42743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:51.168311 18235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:51.168654 18235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:51.174343 18243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:51.174464 18240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:51.174341 18241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:51.174898 18235 server_base.cc:1034] running on GCE node
I20260807 08:46:51.175199 18235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:51.175518 18235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:51.176756 18235 hybrid_clock.cc:648] HybridClock initialized: now 1786092411176736 us; error 40 us; skew 500 ppm
I20260807 08:46:51.178676 18235 webserver.cc:460] Webserver started at http://127.28.108.190:44301/ using document root <none> and password file <none>
I20260807 08:46:51.178942 18235 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:51.178992 18235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:51.179123 18235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:51.180387 18235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "a02fabc70b3041fca793f93c9e5bce33"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "77c540795bbfea16d8714ba9eaf9ef90"
server_key_iv: "babe1a4526e60e6975f169f65233a3ed"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.180775 18235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "a02fabc70b3041fca793f93c9e5bce33"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "77c540795bbfea16d8714ba9eaf9ef90"
server_key_iv: "babe1a4526e60e6975f169f65233a3ed"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.182647 18235 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 08:46:51.183826 18249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:51.184082 18235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:51.184203 18235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "a02fabc70b3041fca793f93c9e5bce33"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "77c540795bbfea16d8714ba9eaf9ef90"
server_key_iv: "babe1a4526e60e6975f169f65233a3ed"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.184311 18235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:51.216807 18235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:51.217162 18235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:51.217288 18235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:51.222421 18235 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:42743
I20260807 08:46:51.222512 18308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:42743 every 8 connection(s)
I20260807 08:46:51.222944 18235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:51.224404 18309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:51.227052 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 18235
I20260807 08:46:51.227181 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:46:51.227408 29106 external_mini_cluster.cc:1428] Setting key 5def6a537195c03cf25b6183c0d3c5ba
I20260807 08:46:51.227934 18309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33: Bootstrap starting.
I20260807 08:46:51.228703 18309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:51.229054 18309 log.cc:824] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:51.229935 18309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33: No bootstrap required, opened a new log
I20260807 08:46:51.231897 18309 raft_consensus.cc:348] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02fabc70b3041fca793f93c9e5bce33" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42743 } }
I20260807 08:46:51.232090 18309 raft_consensus.cc:374] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:51.232141 18309 raft_consensus.cc:729] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02fabc70b3041fca793f93c9e5bce33, State: Initialized, Role: FOLLOWER
I20260807 08:46:51.232299 18309 consensus_queue.cc:260] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02fabc70b3041fca793f93c9e5bce33" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42743 } }
I20260807 08:46:51.232406 18309 raft_consensus.cc:388] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:51.232455 18309 raft_consensus.cc:482] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:51.232506 18309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:51.233405 18309 raft_consensus.cc:504] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02fabc70b3041fca793f93c9e5bce33" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42743 } }
I20260807 08:46:51.233546 18309 leader_election.cc:302] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a02fabc70b3041fca793f93c9e5bce33; no voters:
I20260807 08:46:51.233839 18309 leader_election.cc:288] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:46:51.233875 18314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:51.233990 18314 raft_consensus.cc:686] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [term 1 LEADER]: Becoming Leader. State: Replica: a02fabc70b3041fca793f93c9e5bce33, State: Running, Role: LEADER
I20260807 08:46:51.234086 18314 consensus_queue.cc:237] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02fabc70b3041fca793f93c9e5bce33" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42743 } }
I20260807 08:46:51.234169 18309 sys_catalog.cc:564] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:51.234472 18316 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a02fabc70b3041fca793f93c9e5bce33. Latest consensus state: current_term: 1 leader_uuid: "a02fabc70b3041fca793f93c9e5bce33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02fabc70b3041fca793f93c9e5bce33" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42743 } } }
I20260807 08:46:51.234486 18315 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a02fabc70b3041fca793f93c9e5bce33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02fabc70b3041fca793f93c9e5bce33" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 42743 } } }
I20260807 08:46:51.234532 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.228109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.28.108.190:42743)
0807 08:46:51.228460 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:51.228469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:51.228501 (+ 32us) server_negotiation.cc:407] Connection header received
0807 08:46:51.229119 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:51.229131 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:51.229249 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:51.229464 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:51.229721 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.230831 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.231796 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.232410 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.232578 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.233359 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:51.233364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:51.233364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:51.233696 (+ 332us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:51.233757 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:51.233771 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:51.233859 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:51.234206 (+ 347us) server_negotiation.cc:299] Negotiation successful
0807 08:46:51.234308 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":119,"threads_started":1}
I20260807 08:46:51.234783 18316 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:51.234802 18315 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33 [sys.catalog]: This master's current role is: LEADER
W20260807 08:46:51.236083 18332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:46:51.236145 18332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:46:51.236366 18326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:51.236577 18326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:51.238356 18326 catalog_manager.cc:1347] Generated new cluster ID: 18b76829ca5b40ef8dcd9f71352fdc1b
I20260807 08:46:51.238430 18326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:46:51.264832 18326 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:46:51.265547 18326 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:51.279217 18326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33: Generated new TSK 0
I20260807 08:46:51.279539 18326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:46:51.283499 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.129:0
--local_ip_for_outbound_sockets=127.28.108.129
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42743
--builtin_ntp_servers=127.28.108.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 08:46:51.388600 18336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:51.388823 18336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:51.388881 18336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:51.390311 18336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:51.390388 18336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:51.390434 18336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.129
I20260807 08:46:51.392081 18336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.108.129
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.129
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:51.392418 18336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:51.392808 18336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:51.398802 18344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:51.398830 18346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:51.398797 18342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:51.399510 18336 server_base.cc:1034] running on GCE node
I20260807 08:46:51.399823 18336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:51.400188 18336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:51.401403 18336 hybrid_clock.cc:648] HybridClock initialized: now 1786092411401378 us; error 46 us; skew 500 ppm
I20260807 08:46:51.403469 18336 webserver.cc:460] Webserver started at http://127.28.108.129:46021/ using document root <none> and password file <none>
I20260807 08:46:51.403872 18336 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:51.403951 18336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:51.404131 18336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:51.405905 18336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/data/instance:
uuid: "48fd2018f22641809e87dee1c3ec917f"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "083af2f3cd68e5bd87a610c016921465"
server_key_iv: "b2a6c25f340a90d82484bc8836441120"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.406395 18336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance:
uuid: "48fd2018f22641809e87dee1c3ec917f"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "083af2f3cd68e5bd87a610c016921465"
server_key_iv: "b2a6c25f340a90d82484bc8836441120"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.408674 18336 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 08:46:51.409875 18353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:51.410181 18336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 08:46:51.410298 18336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/wal
uuid: "48fd2018f22641809e87dee1c3ec917f"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "083af2f3cd68e5bd87a610c016921465"
server_key_iv: "b2a6c25f340a90d82484bc8836441120"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.410392 18336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:51.423699 18336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:51.424068 18336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:51.424191 18336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:51.424458 18336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:51.424949 18336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:51.424996 18336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:51.425022 18336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:51.425083 18336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:51.432354 18336 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.129:41521
I20260807 08:46:51.432471 18477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.129:41521 every 8 connection(s)
I20260807 08:46:51.432889 18336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/data/info.pb
I20260807 08:46:51.439262 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.433856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:52881 (local address 127.28.108.190:42743)
0807 08:46:51.433932 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:51.433935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:51.434243 (+ 308us) server_negotiation.cc:407] Connection header received
0807 08:46:51.434944 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:51.434946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:51.434959 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:51.435011 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:51.435874 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.436409 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.437496 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.437782 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.437875 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.438915 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:51.438917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:51.438917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:51.438924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:51.438942 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:51.438946 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:51.439014 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:51.439124 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 08:46:51.439158 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 08:46:51.439512 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.433877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42743 (local address 127.28.108.129:52881)
0807 08:46:51.434089 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:51.434128 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:51.434814 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:51.435085 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:51.435089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:51.435100 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:51.435753 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.435760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.436525 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.436526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.437369 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.437372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.437895 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.437897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.438055 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.438716 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:46:51.438717 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:51.438777 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:46:51.438805 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:51.439016 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:51.439025 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:51.439029 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:46:51.439113 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 08:46:51.439237 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260807 08:46:51.440188 18478 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42743
I20260807 08:46:51.440325 18478 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:51.440558 18478 heartbeater.cc:511] Master 127.28.108.190:42743 requested a full tablet report, sending...
I20260807 08:46:51.441287 18271 ts_manager.cc:194] Registered new tserver with Master: 48fd2018f22641809e87dee1c3ec917f (127.28.108.129:41521)
I20260807 08:46:51.442102 18271 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.129:52881
I20260807 08:46:51.442441 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 18336
I20260807 08:46:51.442574 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-0/wal/instance
I20260807 08:46:51.442816 29106 external_mini_cluster.cc:1428] Setting key 2210d8d9e742cf97ad8c3aea3cb83e4f
I20260807 08:46:51.444296 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.130:0
--local_ip_for_outbound_sockets=127.28.108.130
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42743
--builtin_ntp_servers=127.28.108.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 08:46:51.534188 18481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:51.534536 18481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:51.534567 18481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:51.536026 18481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:51.536110 18481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:51.536157 18481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.130
I20260807 08:46:51.537782 18481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.108.130
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.130
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:51.538131 18481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:51.538525 18481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:51.544639 18487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:51.544628 18490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:51.544628 18488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:51.545176 18481 server_base.cc:1034] running on GCE node
I20260807 08:46:51.545444 18481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:51.545770 18481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:51.546953 18481 hybrid_clock.cc:648] HybridClock initialized: now 1786092411546941 us; error 38 us; skew 500 ppm
I20260807 08:46:51.548460 18481 webserver.cc:460] Webserver started at http://127.28.108.130:42381/ using document root <none> and password file <none>
I20260807 08:46:51.548724 18481 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:51.548838 18481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:51.548949 18481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:51.550174 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/data/instance:
uuid: "7a7abfb9163a4c1786ba9efcfe15e68d"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "7e140ad8450874d5899a33392e9ad184"
server_key_iv: "e06c922a6d78aca92593ff786dbe0652"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.550514 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance:
uuid: "7a7abfb9163a4c1786ba9efcfe15e68d"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "7e140ad8450874d5899a33392e9ad184"
server_key_iv: "e06c922a6d78aca92593ff786dbe0652"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.552183 18481 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 08:46:51.553278 18496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:51.553579 18481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:51.553790 18481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/wal
uuid: "7a7abfb9163a4c1786ba9efcfe15e68d"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "7e140ad8450874d5899a33392e9ad184"
server_key_iv: "e06c922a6d78aca92593ff786dbe0652"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.553892 18481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:51.575423 18481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:51.575827 18481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:51.576007 18481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:51.576289 18481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:51.576689 18481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:51.576752 18481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:51.576799 18481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:51.576824 18481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:51.584471 18481 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.130:33211
I20260807 08:46:51.584532 18623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.130:33211 every 8 connection(s)
I20260807 08:46:51.584980 18481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/data/info.pb
I20260807 08:46:51.591609 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 18481
I20260807 08:46:51.591748 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/wal/instance
I20260807 08:46:51.592039 29106 external_mini_cluster.cc:1428] Setting key 543e20f26f225effa3b0191304b0fbae
I20260807 08:46:51.592190 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.586270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.130:49997 (local address 127.28.108.190:42743)
0807 08:46:51.586359 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:51.586362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:51.586525 (+ 163us) server_negotiation.cc:407] Connection header received
0807 08:46:51.587059 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:51.587060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:51.587073 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:51.587299 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:51.588434 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.589247 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.590503 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.590705 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.590787 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.591828 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:51.591830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:51.591830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:51.591837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:51.591855 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:51.591861 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:51.591950 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:51.592068 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 08:46:51.592102 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 08:46:51.592617 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.586137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42743 (local address 127.28.108.130:49997)
0807 08:46:51.586351 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:51.586397 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:51.586914 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:51.587575 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:51.587580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:51.587590 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:51.588262 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.588271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.589386 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.589387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.590394 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.590397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.590816 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.590817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.590954 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.591591 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:46:51.591592 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:51.591647 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:46:51.591680 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:51.591953 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:51.591962 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:51.591966 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:46:51.592070 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 08:46:51.592345 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260807 08:46:51.593266 18624 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42743
I20260807 08:46:51.593403 18624 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:51.593577 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.108.131:0
--local_ip_for_outbound_sockets=127.28.108.131
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42743
--builtin_ntp_servers=127.28.108.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 08:46:51.593742 18624 heartbeater.cc:511] Master 127.28.108.190:42743 requested a full tablet report, sending...
I20260807 08:46:51.594241 18271 ts_manager.cc:194] Registered new tserver with Master: 7a7abfb9163a4c1786ba9efcfe15e68d (127.28.108.130:33211)
I20260807 08:46:51.594955 18271 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.130:49997
W20260807 08:46:51.691470 18628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:51.691713 18628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:51.691738 18628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:51.692992 18628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:51.693058 18628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:51.693102 18628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.108.131
I20260807 08:46:51.694747 18628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.108.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.108.131
--webserver_port=0
--tserver_master_addrs=127.28.108.190:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.108.131
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:51.695122 18628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:51.695499 18628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:51.702148 18633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:51.702170 18634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:51.702488 18636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:51.702584 18628 server_base.cc:1034] running on GCE node
I20260807 08:46:51.702873 18628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:51.703202 18628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:51.704454 18628 hybrid_clock.cc:648] HybridClock initialized: now 1786092411704435 us; error 53 us; skew 500 ppm
I20260807 08:46:51.706415 18628 webserver.cc:460] Webserver started at http://127.28.108.131:43073/ using document root <none> and password file <none>
I20260807 08:46:51.706754 18628 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:51.706844 18628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:51.707012 18628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:51.708223 18628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/data/instance:
uuid: "e7336cc35afc4215942357108a216b5a"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "bb87ac96f3f8c2ea13e49d3bc6b724fd"
server_key_iv: "5b35e14e31e29f2d144860e63b604ae6"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.708673 18628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance:
uuid: "e7336cc35afc4215942357108a216b5a"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "bb87ac96f3f8c2ea13e49d3bc6b724fd"
server_key_iv: "5b35e14e31e29f2d144860e63b604ae6"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.710469 18628 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 08:46:51.711524 18643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:51.711766 18628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:51.711913 18628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/wal
uuid: "e7336cc35afc4215942357108a216b5a"
format_stamp: "Formatted at 2026-08-07 08:46:51 on dist-test-slave-9zdj"
server_key: "bb87ac96f3f8c2ea13e49d3bc6b724fd"
server_key_iv: "5b35e14e31e29f2d144860e63b604ae6"
server_key_version: "encryptionkey@0"
I20260807 08:46:51.712019 18628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:51.724927 18628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:51.725366 18628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:51.725555 18628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:51.725847 18628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 08:46:51.726233 18628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 08:46:51.726296 18628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:51.726341 18628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 08:46:51.726389 18628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:51.733084 18628 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.131:43233
I20260807 08:46:51.733127 18772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.131:43233 every 8 connection(s)
I20260807 08:46:51.733516 18628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/data/info.pb
I20260807 08:46:51.740602 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.734707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.131:40649 (local address 127.28.108.190:42743)
0807 08:46:51.734849 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:51.734851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:51.735084 (+ 233us) server_negotiation.cc:407] Connection header received
0807 08:46:51.735843 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:51.735845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:51.735859 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:51.735946 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:51.736757 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.737346 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.738750 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.738986 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.739072 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.740157 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:51.740158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:51.740159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:51.740165 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:51.740182 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:51.740186 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:51.740307 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:51.740477 (+ 170us) server_negotiation.cc:299] Negotiation successful
0807 08:46:51.740511 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 08:46:51.740926 18776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.734731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:42743 (local address 127.28.108.131:40649)
0807 08:46:51.734921 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:51.734965 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:51.735645 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:51.736054 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:51.736058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:51.736067 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:46:51.736619 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.736623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.737479 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.737480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.738603 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.738606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.739117 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.739118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.739370 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.739996 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 08:46:51.739996 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:46:51.740030 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 08:46:51.740058 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:46:51.740352 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:46:51.740360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:46:51.740363 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 08:46:51.740452 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 08:46:51.740566 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260807 08:46:51.741416 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 18628
I20260807 08:46:51.741541 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/wal/instance
I20260807 08:46:51.741585 18773 heartbeater.cc:347] Connected to a master server at 127.28.108.190:42743
I20260807 08:46:51.741730 18773 heartbeater.cc:464] Registering TS with master...
I20260807 08:46:51.741811 29106 external_mini_cluster.cc:1428] Setting key 91ad86bcd9d2e8c039ceb711ec9d0ed7
I20260807 08:46:51.742000 18773 heartbeater.cc:511] Master 127.28.108.190:42743 requested a full tablet report, sending...
I20260807 08:46:51.742424 18271 ts_manager.cc:194] Registered new tserver with Master: e7336cc35afc4215942357108a216b5a (127.28.108.131:43233)
I20260807 08:46:51.743103 18271 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.108.131:40649
I20260807 08:46:51.743345 29106 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 08:46:51.748399 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.745097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.28.108.190:42743)
0807 08:46:51.745257 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:51.745259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:51.745361 (+ 102us) server_negotiation.cc:407] Connection header received
0807 08:46:51.745433 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:51.745434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:51.745445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:51.745498 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:51.745723 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.746237 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.747036 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.747255 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.747341 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.748140 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:51.748141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:51.748142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:51.748148 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:51.748165 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:51.748169 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:51.748218 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:51.748275 (+ 57us) server_negotiation.cc:299] Negotiation successful
0807 08:46:51.748303 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260807 08:46:51.750777 18271 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 08:46:51.752416 18271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 08:46:51.760586 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.755258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:33211 (local address 127.0.0.1:50414)
0807 08:46:51.755836 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:51.755844 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:51.755909 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:51.756293 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:51.756296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:51.756298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:51.756403 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.756404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.757990 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.757991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.759073 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.759077 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.760157 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.760158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.760256 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.760272 (+ 16us) client_negotiation.cc:769] Sending connection context
0807 08:46:51.760395 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 08:46:51.760474 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":6,"thread_start_us":86,"threads_started":1}
I20260807 08:46:51.760633 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.755450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.28.108.130:33211)
0807 08:46:51.755792 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:51.755797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:51.755892 (+ 95us) server_negotiation.cc:407] Connection header received
0807 08:46:51.756053 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:51.756056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:51.756075 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:51.756176 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:51.756498 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.757864 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.759201 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.760020 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.760122 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.760252 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:51.760438 (+ 186us) server_negotiation.cc:299] Negotiation successful
0807 08:46:51.760509 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":118,"threads_started":1}
I20260807 08:46:51.761013 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.755251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:43233 (local address 127.0.0.1:59798)
0807 08:46:51.755667 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:51.755719 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:51.755980 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:51.756294 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:51.756296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:51.756302 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:51.756395 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.756398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.758020 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.758021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.759363 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.759367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.760763 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.760764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.760834 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.760839 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 08:46:51.760896 (+ 57us) client_negotiation.cc:240] Negotiation successful
0807 08:46:51.760945 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":16,"thread_start_us":80,"threads_started":1}
I20260807 08:46:51.761066 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.755457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.28.108.131:43233)
0807 08:46:51.755686 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:51.755692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:51.755795 (+ 103us) server_negotiation.cc:407] Connection header received
0807 08:46:51.756103 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:51.756106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:51.756123 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:51.756200 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:51.756496 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.757889 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.759479 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.760322 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.760443 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.760568 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:51.760922 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 08:46:51.760972 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
I20260807 08:46:51.761899 18700 tablet_service.cc:1467] Processing CreateTablet for tablet 964b7adb661b46b0b747a6a986917c12 (DEFAULT_TABLE table=test-table [id=174bc26254154a9c83b17aea7be0c847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:51.761880 18548 tablet_service.cc:1467] Processing CreateTablet for tablet 964b7adb661b46b0b747a6a986917c12 (DEFAULT_TABLE table=test-table [id=174bc26254154a9c83b17aea7be0c847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:51.762367 18700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964b7adb661b46b0b747a6a986917c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:51.762367 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.755568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.28.108.129:41521)
0807 08:46:51.755766 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:51.755771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:51.756205 (+ 434us) server_negotiation.cc:407] Connection header received
0807 08:46:51.756258 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:51.756260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:51.756323 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:51.756403 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:51.756565 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.757591 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.760628 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.761896 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.762001 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.762100 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:51.762195 (+ 95us) server_negotiation.cc:299] Negotiation successful
0807 08:46:51.762246 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260807 08:46:51.762377 18548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964b7adb661b46b0b747a6a986917c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:51.762292 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.755338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:41521 (local address 127.0.0.1:36582)
0807 08:46:51.756081 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:51.756090 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:51.756134 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:51.756422 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:51.756422 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:51.756424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:51.756477 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.756479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.757717 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.757719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.759363 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.759367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.762015 (+ 2648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.762017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.762093 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.762097 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 08:46:51.762154 (+ 57us) client_negotiation.cc:240] Negotiation successful
0807 08:46:51.762209 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":81,"thread_start_us":42,"threads_started":1}
I20260807 08:46:51.763368 18404 tablet_service.cc:1467] Processing CreateTablet for tablet 964b7adb661b46b0b747a6a986917c12 (DEFAULT_TABLE table=test-table [id=174bc26254154a9c83b17aea7be0c847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 08:46:51.763815 18404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964b7adb661b46b0b747a6a986917c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:51.765544 18791 tablet_bootstrap.cc:492] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a: Bootstrap starting.
I20260807 08:46:51.766255 18791 tablet_bootstrap.cc:654] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:51.766453 18793 tablet_bootstrap.cc:492] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: Bootstrap starting.
I20260807 08:46:51.766651 18791 log.cc:824] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:51.767190 18793 tablet_bootstrap.cc:654] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:51.767563 18793 log.cc:824] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:51.768033 18791 tablet_bootstrap.cc:492] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a: No bootstrap required, opened a new log
I20260807 08:46:51.768088 18796 tablet_bootstrap.cc:492] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f: Bootstrap starting.
I20260807 08:46:51.768129 18791 ts_tablet_manager.cc:1397] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 08:46:51.768509 18793 tablet_bootstrap.cc:492] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: No bootstrap required, opened a new log
I20260807 08:46:51.768623 18793 ts_tablet_manager.cc:1397] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 08:46:51.768783 18796 tablet_bootstrap.cc:654] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:51.769145 18796 log.cc:824] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:51.769971 18796 tablet_bootstrap.cc:492] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f: No bootstrap required, opened a new log
I20260807 08:46:51.770032 18796 ts_tablet_manager.cc:1397] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 08:46:51.771102 18791 raft_consensus.cc:348] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7336cc35afc4215942357108a216b5a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43233 } } peers { permanent_uuid: "48fd2018f22641809e87dee1c3ec917f" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41521 } } peers { permanent_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 33211 } }
I20260807 08:46:51.771353 18791 raft_consensus.cc:374] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:51.771406 18791 raft_consensus.cc:729] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7336cc35afc4215942357108a216b5a, State: Initialized, Role: FOLLOWER
I20260807 08:46:51.771559 18791 consensus_queue.cc:260] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7336cc35afc4215942357108a216b5a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43233 } } peers { permanent_uuid: "48fd2018f22641809e87dee1c3ec917f" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41521 } } peers { permanent_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 33211 } }
I20260807 08:46:51.771880 18791 ts_tablet_manager.cc:1428] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 08:46:51.771956 18773 heartbeater.cc:503] Master 127.28.108.190:42743 was elected leader, sending a full tablet report...
I20260807 08:46:51.771977 18793 raft_consensus.cc:348] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7336cc35afc4215942357108a216b5a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43233 } } peers { permanent_uuid: "48fd2018f22641809e87dee1c3ec917f" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41521 } } peers { permanent_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 33211 } }
I20260807 08:46:51.772182 18793 raft_consensus.cc:374] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:51.772229 18793 raft_consensus.cc:729] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7abfb9163a4c1786ba9efcfe15e68d, State: Initialized, Role: FOLLOWER
I20260807 08:46:51.772358 18793 consensus_queue.cc:260] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7336cc35afc4215942357108a216b5a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43233 } } peers { permanent_uuid: "48fd2018f22641809e87dee1c3ec917f" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41521 } } peers { permanent_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 33211 } }
I20260807 08:46:51.772702 18793 ts_tablet_manager.cc:1428] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260807 08:46:51.772732 18624 heartbeater.cc:503] Master 127.28.108.190:42743 was elected leader, sending a full tablet report...
I20260807 08:46:51.772830 18796 raft_consensus.cc:348] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7336cc35afc4215942357108a216b5a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43233 } } peers { permanent_uuid: "48fd2018f22641809e87dee1c3ec917f" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41521 } } peers { permanent_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 33211 } }
I20260807 08:46:51.773020 18796 raft_consensus.cc:374] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:51.773082 18796 raft_consensus.cc:729] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48fd2018f22641809e87dee1c3ec917f, State: Initialized, Role: FOLLOWER
I20260807 08:46:51.773309 18796 consensus_queue.cc:260] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7336cc35afc4215942357108a216b5a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43233 } } peers { permanent_uuid: "48fd2018f22641809e87dee1c3ec917f" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41521 } } peers { permanent_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 33211 } }
I20260807 08:46:51.773689 18796 ts_tablet_manager.cc:1428] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 08:46:51.773828 18478 heartbeater.cc:503] Master 127.28.108.190:42743 was elected leader, sending a full tablet report...
W20260807 08:46:51.835788 18625 tablet.cc:2369] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:46:51.856132 18801 raft_consensus.cc:482] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 08:46:51.856285 18801 raft_consensus.cc:504] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7336cc35afc4215942357108a216b5a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43233 } } peers { permanent_uuid: "48fd2018f22641809e87dee1c3ec917f" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41521 } } peers { permanent_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 33211 } }
I20260807 08:46:51.856760 18801 leader_election.cc:288] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7336cc35afc4215942357108a216b5a (127.28.108.131:43233), 7a7abfb9163a4c1786ba9efcfe15e68d (127.28.108.130:33211)
I20260807 08:46:51.861443 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.857168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:46231 (local address 127.28.108.130:33211)
0807 08:46:51.857213 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:51.857216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:51.857337 (+ 121us) server_negotiation.cc:407] Connection header received
0807 08:46:51.857393 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:51.857395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:51.857408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:51.857472 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:51.857686 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.858467 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.860325 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.860914 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.860992 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.861000 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:51.861313 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 08:46:51.861365 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:46:51.861442 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.857167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.108.129:38303 (local address 127.28.108.131:43233)
0807 08:46:51.857213 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:51.857216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:51.857227 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:46:51.857264 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:51.857270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:51.857279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:51.857383 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 08:46:51.857638 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.858409 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.860214 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.860851 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.860952 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.860965 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:51.861305 (+ 340us) server_negotiation.cc:299] Negotiation successful
0807 08:46:51.861365 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 08:46:51.861409 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.857027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.131:43233 (local address 127.28.108.129:38303)
0807 08:46:51.857098 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:51.857122 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:51.857179 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:51.857391 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:51.857392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:51.857394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:51.857512 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.857515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.858567 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.858569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.860057 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.860060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.861036 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.861038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.861139 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.861148 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 08:46:51.861230 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 08:46:51.861303 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":18}
I20260807 08:46:51.861415 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.857041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.130:33211 (local address 127.28.108.129:46231)
0807 08:46:51.857249 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:51.857260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:46:51.857328 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:46:51.857521 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:46:51.857522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:46:51.857524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 08:46:51.857596 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.857598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.858567 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.858569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.860056 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:46:51.860060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.861050 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.861051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:46:51.861139 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.861149 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 08:46:51.861237 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 08:46:51.861294 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":93,"thread_start_us":105,"threads_started":1}
I20260807 08:46:51.861953 18721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964b7adb661b46b0b747a6a986917c12" candidate_uuid: "48fd2018f22641809e87dee1c3ec917f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7336cc35afc4215942357108a216b5a" is_pre_election: true
I20260807 08:46:51.861953 18570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964b7adb661b46b0b747a6a986917c12" candidate_uuid: "48fd2018f22641809e87dee1c3ec917f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" is_pre_election: true
I20260807 08:46:51.862157 18721 raft_consensus.cc:2454] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48fd2018f22641809e87dee1c3ec917f in term 0.
I20260807 08:46:51.862157 18570 raft_consensus.cc:2454] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48fd2018f22641809e87dee1c3ec917f in term 0.
I20260807 08:46:51.862627 18355 leader_election.cc:302] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48fd2018f22641809e87dee1c3ec917f, e7336cc35afc4215942357108a216b5a; no voters:
I20260807 08:46:51.862811 18801 raft_consensus.cc:2781] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 08:46:51.862912 18801 raft_consensus.cc:482] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 08:46:51.862934 18801 raft_consensus.cc:3037] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:51.863895 18801 raft_consensus.cc:504] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7336cc35afc4215942357108a216b5a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43233 } } peers { permanent_uuid: "48fd2018f22641809e87dee1c3ec917f" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41521 } } peers { permanent_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 33211 } }
I20260807 08:46:51.864069 18801 leader_election.cc:288] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [CANDIDATE]: Term 1 election: Requested vote from peers e7336cc35afc4215942357108a216b5a (127.28.108.131:43233), 7a7abfb9163a4c1786ba9efcfe15e68d (127.28.108.130:33211)
I20260807 08:46:51.864307 18570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964b7adb661b46b0b747a6a986917c12" candidate_uuid: "48fd2018f22641809e87dee1c3ec917f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d"
I20260807 08:46:51.864456 18570 raft_consensus.cc:3037] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:51.864535 18721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964b7adb661b46b0b747a6a986917c12" candidate_uuid: "48fd2018f22641809e87dee1c3ec917f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7336cc35afc4215942357108a216b5a"
I20260807 08:46:51.864670 18721 raft_consensus.cc:3037] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:51.865474 18570 raft_consensus.cc:2454] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48fd2018f22641809e87dee1c3ec917f in term 1.
I20260807 08:46:51.865566 18721 raft_consensus.cc:2454] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48fd2018f22641809e87dee1c3ec917f in term 1.
I20260807 08:46:51.865748 18356 leader_election.cc:302] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48fd2018f22641809e87dee1c3ec917f, 7a7abfb9163a4c1786ba9efcfe15e68d; no voters:
I20260807 08:46:51.865866 18801 raft_consensus.cc:2781] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:51.866070 18801 raft_consensus.cc:686] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [term 1 LEADER]: Becoming Leader. State: Replica: 48fd2018f22641809e87dee1c3ec917f, State: Running, Role: LEADER
I20260807 08:46:51.866211 18801 consensus_queue.cc:237] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7336cc35afc4215942357108a216b5a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43233 } } peers { permanent_uuid: "48fd2018f22641809e87dee1c3ec917f" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41521 } } peers { permanent_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 33211 } }
I20260807 08:46:51.867434 18271 catalog_manager.cc:5495] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f reported cstate change: term changed from 0 to 1, leader changed from <none> to 48fd2018f22641809e87dee1c3ec917f (127.28.108.129). New cstate: current_term: 1 leader_uuid: "48fd2018f22641809e87dee1c3ec917f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7336cc35afc4215942357108a216b5a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48fd2018f22641809e87dee1c3ec917f" member_type: VOTER last_known_addr { host: "127.28.108.129" port: 41521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20260807 08:46:51.878739 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.875879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.28.108.129:41521)
0807 08:46:51.875953 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:51.875955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:51.875965 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:46:51.876010 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:51.876011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:51.876034 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:51.876113 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:46:51.876289 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.876855 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.877752 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:51.877957 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:51.878016 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:51.878196 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:46:51.878315 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:46:51.878460 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:51.878564 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 08:46:51.878590 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 08:46:51.880916 18721 raft_consensus.cc:1261] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a [term 1 FOLLOWER]: Refusing update from remote peer 48fd2018f22641809e87dee1c3ec917f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:51.880901 18570 raft_consensus.cc:1261] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d [term 1 FOLLOWER]: Refusing update from remote peer 48fd2018f22641809e87dee1c3ec917f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:46:51.881429 18803 consensus_queue.cc:1035] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7336cc35afc4215942357108a216b5a" member_type: VOTER last_known_addr { host: "127.28.108.131" port: 43233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:51.881588 18801 consensus_queue.cc:1035] T 964b7adb661b46b0b747a6a986917c12 P 48fd2018f22641809e87dee1c3ec917f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7abfb9163a4c1786ba9efcfe15e68d" member_type: VOTER last_known_addr { host: "127.28.108.130" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:46:51.884828 18807 mvcc.cc:205] Tried to move back new op lower bound from 7315834519061069824 to 7315834519004913664. Current Snapshot: MvccSnapshot[applied={T|T < 7315834519061069824}]
I20260807 08:46:51.885066 18806 mvcc.cc:205] Tried to move back new op lower bound from 7315834519061069824 to 7315834519004913664. Current Snapshot: MvccSnapshot[applied={T|T < 7315834519061069824}]
I20260807 08:46:51.888059 18271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:51.888191 18271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 08:46:51.889128 18271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a02fabc70b3041fca793f93c9e5bce33: Sending DeleteTablet for 3 replicas of tablet 964b7adb661b46b0b747a6a986917c12
I20260807 08:46:51.889554 18700 tablet_service.cc:1514] Processing DeleteTablet for tablet 964b7adb661b46b0b747a6a986917c12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:51 UTC) from {username='slave'} at 127.0.0.1:59798
I20260807 08:46:51.889590 18548 tablet_service.cc:1514] Processing DeleteTablet for tablet 964b7adb661b46b0b747a6a986917c12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:51 UTC) from {username='slave'} at 127.0.0.1:50414
I20260807 08:46:51.889642 18404 tablet_service.cc:1514] Processing DeleteTablet for tablet 964b7adb661b46b0b747a6a986917c12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 08:46:51 UTC) from {username='slave'} at 127.0.0.1:36582
I20260807 08:46:51.890013 18818 tablet_replica.cc:330] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: stopping tablet replica
I20260807 08:46:51.890164 18818 raft_consensus.cc:2229] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:51.890290 18818 raft_consensus.cc:2258] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:51.890314 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 18336
I20260807 08:46:51.895155 18817 tablet_replica.cc:330] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a: stopping tablet replica
I20260807 08:46:51.895364 18817 raft_consensus.cc:2229] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 08:46:51.895498 18817 raft_consensus.cc:2258] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 08:46:51.895608 18818 ts_tablet_manager.cc:1905] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:51.896092 18817 ts_tablet_manager.cc:1905] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 08:46:51.897662 18818 ts_tablet_manager.cc:1918] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:51.897915 18818 log.cc:1196] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-1/wal/wals/964b7adb661b46b0b747a6a986917c12
I20260807 08:46:51.898200 18818 ts_tablet_manager.cc:1939] T 964b7adb661b46b0b747a6a986917c12 P 7a7abfb9163a4c1786ba9efcfe15e68d: Deleting consensus metadata
I20260807 08:46:51.898779 18252 catalog_manager.cc:4841] TS 7a7abfb9163a4c1786ba9efcfe15e68d (127.28.108.130:33211): tablet 964b7adb661b46b0b747a6a986917c12 (table test-table [id=174bc26254154a9c83b17aea7be0c847]) successfully deleted
I20260807 08:46:51.900758 18817 ts_tablet_manager.cc:1918] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 08:46:51.900892 18817 log.cc:1196] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a: Deleting WAL directory at /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786092133523677-29106-0/minicluster-data/ts-2/wal/wals/964b7adb661b46b0b747a6a986917c12
W20260807 08:46:51.901144 18250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.108.129:41521 (error 108)
I20260807 08:46:51.901407 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 18481
I20260807 08:46:51.902176 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:51.901717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.129:41521 (local address 127.0.0.1:36598)
0807 08:46:51.902078 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0807 08:46:51.902133 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.129:41521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330}
W20260807 08:46:51.902875 18250 catalog_manager.cc:4584] TS 48fd2018f22641809e87dee1c3ec917f (127.28.108.129:41521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 964b7adb661b46b0b747a6a986917c12: Network error: Client connection negotiation failed: client connection to 127.28.108.129:41521: connect: Connection refused (error 111)
I20260807 08:46:51.903183 18817 ts_tablet_manager.cc:1939] T 964b7adb661b46b0b747a6a986917c12 P e7336cc35afc4215942357108a216b5a: Deleting consensus metadata
I20260807 08:46:51.903757 18251 catalog_manager.cc:4841] TS e7336cc35afc4215942357108a216b5a (127.28.108.131:43233): tablet 964b7adb661b46b0b747a6a986917c12 (table test-table [id=174bc26254154a9c83b17aea7be0c847]) successfully deleted
I20260807 08:46:51.910745 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 18628
I20260807 08:46:51.919409 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 18235
2026-08-07T08:46:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (886 ms)
[----------] 6 tests from EncryptionPolicyTest (10053 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:46:51 dist-test-slave-9zdj krb5kdc[18826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:46:51 dist-test-slave-9zdj krb5kdc[18826](info): set up 2 sockets
Aug 07 08:46:51 dist-test-slave-9zdj krb5kdc[18826](info): commencing operation
krb5kdc: starting...
W20260807 08:46:53.067847 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:46:53 dist-test-slave-9zdj krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:46:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:46:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:46:53.218374 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:53.315836 18842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:53.316098 18842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:53.316128 18842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:53.317456 18842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:53.317510 18842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:53.317528 18842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:53.317543 18842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:53.317556 18842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:53.319392 18842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:53.319639 18842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:53.319839 18842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:53.325711 18847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:53.325711 18849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:53.325745 18851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:53.325958 18842 server_base.cc:1034] running on GCE node
I20260807 08:46:53.326309 18842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:53.326699 18842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:53.327898 18842 hybrid_clock.cc:648] HybridClock initialized: now 1786092413327864 us; error 59 us; skew 500 ppm
Aug 07 08:46:53 dist-test-slave-9zdj krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:53.331023 18842 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:46:53.331537 18842 webserver.cc:460] Webserver started at http://127.0.0.1:44755/ using document root <none> and password file <none>
I20260807 08:46:53.331809 18842 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:53.331887 18842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:53.332043 18842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:53.333202 18842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "159ee86f50024e19a83eecc90a4d3df0"
format_stamp: "Formatted at 2026-08-07 08:46:53 on dist-test-slave-9zdj"
server_key: "d4fe91718a006a115d45296472c76599"
server_key_iv: "b1580a24b1619823253238e34f30ff91"
server_key_version: "encryptionkey@0"
I20260807 08:46:53.333604 18842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "159ee86f50024e19a83eecc90a4d3df0"
format_stamp: "Formatted at 2026-08-07 08:46:53 on dist-test-slave-9zdj"
server_key: "d4fe91718a006a115d45296472c76599"
server_key_iv: "b1580a24b1619823253238e34f30ff91"
server_key_version: "encryptionkey@0"
I20260807 08:46:53.335314 18842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:46:53.336236 18859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:53.336524 18842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:53.336655 18842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "159ee86f50024e19a83eecc90a4d3df0"
format_stamp: "Formatted at 2026-08-07 08:46:53 on dist-test-slave-9zdj"
server_key: "d4fe91718a006a115d45296472c76599"
server_key_iv: "b1580a24b1619823253238e34f30ff91"
server_key_version: "encryptionkey@0"
I20260807 08:46:53.336741 18842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:53.348742 18842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:53.351693 18842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:53.351994 18842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:53.357189 18842 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45167
I20260807 08:46:53.357316 18919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45167 every 8 connection(s)
I20260807 08:46:53.357753 18842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:53.358734 18920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:53.361413 18920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0: Bootstrap starting.
I20260807 08:46:53.362207 18920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:53.362509 18920 log.cc:824] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:53.363368 18920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0: No bootstrap required, opened a new log
I20260807 08:46:53.365722 18920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159ee86f50024e19a83eecc90a4d3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260807 08:46:53.365979 18920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:53.366036 18920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159ee86f50024e19a83eecc90a4d3df0, State: Initialized, Role: FOLLOWER
I20260807 08:46:53.366225 18920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159ee86f50024e19a83eecc90a4d3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260807 08:46:53.366331 18920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:53.366374 18920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:53.366379 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 18842
I20260807 08:46:53.366428 18920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:53.366506 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:46:53.366724 29106 external_mini_cluster.cc:1428] Setting key fed4bb5ba02a403b776f034e58ed4fb3
I20260807 08:46:53.367467 18920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159ee86f50024e19a83eecc90a4d3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260807 08:46:53.367604 18920 leader_election.cc:302] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 159ee86f50024e19a83eecc90a4d3df0; no voters:
I20260807 08:46:53.367805 18920 leader_election.cc:288] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:46:53.367843 18926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:53.368007 18926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [term 1 LEADER]: Becoming Leader. State: Replica: 159ee86f50024e19a83eecc90a4d3df0, State: Running, Role: LEADER
I20260807 08:46:53.368124 18926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159ee86f50024e19a83eecc90a4d3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260807 08:46:53.368155 18920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:53.368527 18926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 159ee86f50024e19a83eecc90a4d3df0. Latest consensus state: current_term: 1 leader_uuid: "159ee86f50024e19a83eecc90a4d3df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159ee86f50024e19a83eecc90a4d3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } } }
I20260807 08:46:53.368616 18926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:53.368857 18927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "159ee86f50024e19a83eecc90a4d3df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159ee86f50024e19a83eecc90a4d3df0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } } }
I20260807 08:46:53.368980 18927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0 [sys.catalog]: This master's current role is: LEADER
W20260807 08:46:53.370630 18946 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:46:53.370715 18946 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:46:53.370779 18947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:53.370962 18947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:53.375231 18947 catalog_manager.cc:1347] Generated new cluster ID: 478f8833607b4e8eb5cdf1beb65e0f74
I20260807 08:46:53.375319 18947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:46:53 dist-test-slave-9zdj krb5kdc[18826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 08:46:53.381278 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:53.367357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53675 (local address 127.0.0.1:45167)
0807 08:46:53.367630 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:53.367643 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:53.367687 (+ 44us) server_negotiation.cc:407] Connection header received
0807 08:46:53.367860 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:53.367868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:53.367942 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:53.369397 (+ 1455us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:53.369532 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:53.371360 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:53.372530 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:53.373161 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:53.373289 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:53.376415 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:53.376420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:53.376421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:53.376438 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:53.379111 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:53.379689 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:53.379693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:53.379694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:53.379855 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:53.380146 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:53.380147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:53.380148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:53.380545 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:53.380745 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:53.380869 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 08:46:53.381015 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":130,"threads_started":1}
I20260807 08:46:53.382926 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:46:53.386714 18947 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:46:53.387252 18947 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:53.390027 18947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 159ee86f50024e19a83eecc90a4d3df0: Generated new TSK 0
I20260807 08:46:53.390199 18947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:46:53.393371 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:53.384735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42331 (local address 127.0.0.1:45167)
0807 08:46:53.384776 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:53.384777 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:53.384787 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:46:53.384851 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:53.384852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:53.384862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:53.384923 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:53.385763 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:53.386591 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:53.387890 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:53.388278 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:53.388323 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:53.390090 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:53.390091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:53.390091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:53.390100 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:53.391900 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:53.392456 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:53.392457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:53.392458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:53.392527 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:53.392787 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:53.392788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:53.392788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:53.392941 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:53.393050 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:53.393194 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 08:46:53.393269 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":21}
I20260807 08:46:53.396157 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 18842
W20260807 08:46:53.405504 18822 connection.cc:535] client connection to 127.0.0.1:45167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45167): Connection reset by peer (error 104)
2026-08-07T08:46:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:46:53 dist-test-slave-9zdj krb5kdc[18958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:46:53 dist-test-slave-9zdj krb5kdc[18958](info): set up 2 sockets
Aug 07 08:46:53 dist-test-slave-9zdj krb5kdc[18958](info): commencing operation
krb5kdc: starting...
W20260807 08:46:54.584745 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:46:54 dist-test-slave-9zdj krb5kdc[18958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:46:54.738014 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:54.832212 18972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:54.832527 18972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:54.832576 18972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:54.833925 18972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:54.834010 18972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:54.834036 18972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:54.834059 18972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:54.834089 18972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:54.835704 18972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34519
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:54.836028 18972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:54.836301 18972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:54.841989 18979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:54.842058 18978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:54.842082 18972 server_base.cc:1034] running on GCE node
W20260807 08:46:54.841989 18981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:54.842561 18972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:54.842964 18972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:54.844148 18972 hybrid_clock.cc:648] HybridClock initialized: now 1786092414844106 us; error 63 us; skew 500 ppm
Aug 07 08:46:54 dist-test-slave-9zdj krb5kdc[18958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:54.847818 18972 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:46:54.848392 18972 webserver.cc:460] Webserver started at http://127.0.0.1:33501/ using document root <none> and password file <none>
I20260807 08:46:54.848680 18972 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:54.848763 18972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:54.848935 18972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:54.850378 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "0a6faf0b9617409b851a46fbde2151e3"
format_stamp: "Formatted at 2026-08-07 08:46:54 on dist-test-slave-9zdj"
server_key: "03d9f7fa8089d8fec5000dd939245db7"
server_key_iv: "000dce03bfa6cee4743c7a94ebf6f20d"
server_key_version: "encryptionkey@0"
I20260807 08:46:54.850901 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "0a6faf0b9617409b851a46fbde2151e3"
format_stamp: "Formatted at 2026-08-07 08:46:54 on dist-test-slave-9zdj"
server_key: "03d9f7fa8089d8fec5000dd939245db7"
server_key_iv: "000dce03bfa6cee4743c7a94ebf6f20d"
server_key_version: "encryptionkey@0"
I20260807 08:46:54.852742 18972 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 08:46:54.853870 18989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:54.854099 18972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:54.854234 18972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "0a6faf0b9617409b851a46fbde2151e3"
format_stamp: "Formatted at 2026-08-07 08:46:54 on dist-test-slave-9zdj"
server_key: "03d9f7fa8089d8fec5000dd939245db7"
server_key_iv: "000dce03bfa6cee4743c7a94ebf6f20d"
server_key_version: "encryptionkey@0"
I20260807 08:46:54.854328 18972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:54.866374 18972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:54.869823 18972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:54.870009 18972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:54.874909 18972 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34519
I20260807 08:46:54.874970 19048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34519 every 8 connection(s)
I20260807 08:46:54.875424 18972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:54.876571 19050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:54.879422 19050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3: Bootstrap starting.
I20260807 08:46:54.880148 19050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:54.880507 19050 log.cc:824] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:54.881470 19050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3: No bootstrap required, opened a new log
I20260807 08:46:54.883430 19050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6faf0b9617409b851a46fbde2151e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260807 08:46:54.883599 19050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:54.883626 19050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6faf0b9617409b851a46fbde2151e3, State: Initialized, Role: FOLLOWER
I20260807 08:46:54.883813 19050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6faf0b9617409b851a46fbde2151e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260807 08:46:54.883889 19050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:54.883915 19050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:54.883972 19050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:54.884658 19050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6faf0b9617409b851a46fbde2151e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260807 08:46:54.884793 19050 leader_election.cc:302] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a6faf0b9617409b851a46fbde2151e3; no voters:
I20260807 08:46:54.885064 19050 leader_election.cc:288] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:46:54.885154 19053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:54.885481 19053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [term 1 LEADER]: Becoming Leader. State: Replica: 0a6faf0b9617409b851a46fbde2151e3, State: Running, Role: LEADER
I20260807 08:46:54.885597 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 18972
I20260807 08:46:54.885754 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:46:54.885754 19053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6faf0b9617409b851a46fbde2151e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260807 08:46:54.885892 19050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:54.886025 29106 external_mini_cluster.cc:1428] Setting key 29f3ddd0aaa3f2d4ef2a27f3130e779d
I20260807 08:46:54.886567 19054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a6faf0b9617409b851a46fbde2151e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6faf0b9617409b851a46fbde2151e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } } }
I20260807 08:46:54.886654 19054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:54.887145 19055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a6faf0b9617409b851a46fbde2151e3. Latest consensus state: current_term: 1 leader_uuid: "0a6faf0b9617409b851a46fbde2151e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6faf0b9617409b851a46fbde2151e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } } }
I20260807 08:46:54.887276 19055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:54.887480 19063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:54.888118 19063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:54.890027 19063 catalog_manager.cc:1347] Generated new cluster ID: 8cc0bd85f9694cd98f7b5600e6f6145c
I20260807 08:46:54.890089 19063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:46:54.893167 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:54.886810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40945 (local address 127.0.0.1:34519)
0807 08:46:54.887222 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:54.887234 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:54.887277 (+ 43us) server_negotiation.cc:407] Connection header received
0807 08:46:54.887417 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:54.887426 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:54.887503 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:54.887942 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:54.888081 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:54.889513 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:54.890636 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:54.891129 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:54.891276 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:54.892150 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:54.892155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:54.892156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:54.892554 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:54.892594 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:54.892608 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:54.892758 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:54.892847 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 08:46:54.892881 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":98,"threads_started":1}
I20260807 08:46:54.894344 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:46:54.901014 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:54.895849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49683 (local address 127.0.0.1:34519)
0807 08:46:54.895889 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:54.895891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:54.897959 (+ 2068us) server_negotiation.cc:407] Connection header received
0807 08:46:54.898034 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:54.898036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:54.898051 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:54.898104 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:54.898310 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:54.898854 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:54.899645 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:54.899885 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:54.899969 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:54.900734 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:54.900735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:54.900736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:54.900741 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:54.900756 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:54.900761 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:54.900818 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:54.900891 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 08:46:54.900904 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260807 08:46:54.916347 19063 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:46:54.917182 19063 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:54.928159 19063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a6faf0b9617409b851a46fbde2151e3: Generated new TSK 0
I20260807 08:46:54.928431 19063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:46:54.937414 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 18972
W20260807 08:46:54.945443 18956 connection.cc:535] client connection to 127.0.0.1:34519 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34519): Connection reset by peer (error 104)
2026-08-07T08:46:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1539 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:46:55 dist-test-slave-9zdj krb5kdc[19081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:46:55 dist-test-slave-9zdj krb5kdc[19081](info): set up 2 sockets
Aug 07 08:46:55 dist-test-slave-9zdj krb5kdc[19081](info): commencing operation
krb5kdc: starting...
W20260807 08:46:56.096829 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:46:56 dist-test-slave-9zdj krb5kdc[19081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:46:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:46:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:46:56.263835 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:56.356076 19098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:56.356349 19098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:56.356377 19098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:56.357685 19098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:56.357743 19098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:56.357757 19098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:56.357770 19098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:56.357784 19098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:56.359479 19098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:56.359740 19098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:56.359949 19098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:46:56.366070 19098 server_base.cc:1034] running on GCE node
W20260807 08:46:56.366017 19104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:56.366024 19108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:56.366017 19105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:56.366698 19098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:56.367079 19098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:56.368255 19098 hybrid_clock.cc:648] HybridClock initialized: now 1786092416368244 us; error 35 us; skew 500 ppm
Aug 07 08:46:56 dist-test-slave-9zdj krb5kdc[19081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:56.377254 19098 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:46:56.377959 19098 webserver.cc:460] Webserver started at http://127.0.0.1:45409/ using document root <none> and password file <none>
I20260807 08:46:56.378288 19098 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:56.378346 19098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:56.378507 19098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:56.379745 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "bc7a92ac74df42ef8193dc6ec00bfa74"
format_stamp: "Formatted at 2026-08-07 08:46:56 on dist-test-slave-9zdj"
server_key: "dc5f6c168376686c386dc81b654dfc1e"
server_key_iv: "d5f3eee2573ee30ebc43f1f9dcf20882"
server_key_version: "encryptionkey@0"
I20260807 08:46:56.380193 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "bc7a92ac74df42ef8193dc6ec00bfa74"
format_stamp: "Formatted at 2026-08-07 08:46:56 on dist-test-slave-9zdj"
server_key: "dc5f6c168376686c386dc81b654dfc1e"
server_key_iv: "d5f3eee2573ee30ebc43f1f9dcf20882"
server_key_version: "encryptionkey@0"
I20260807 08:46:56.382115 19098 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 08:46:56.383354 19116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:56.383679 19098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:56.383777 19098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "bc7a92ac74df42ef8193dc6ec00bfa74"
format_stamp: "Formatted at 2026-08-07 08:46:56 on dist-test-slave-9zdj"
server_key: "dc5f6c168376686c386dc81b654dfc1e"
server_key_iv: "d5f3eee2573ee30ebc43f1f9dcf20882"
server_key_version: "encryptionkey@0"
I20260807 08:46:56.383898 19098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:56.387985 19098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:56.391060 19098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:56.391233 19098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:56.396359 19098 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34155
I20260807 08:46:56.396418 19173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34155 every 8 connection(s)
I20260807 08:46:56.396843 19098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:56.398098 19174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:56.400660 19174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74: Bootstrap starting.
I20260807 08:46:56.401269 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 19098
I20260807 08:46:56.401443 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:46:56.401433 19174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:56.401712 29106 external_mini_cluster.cc:1428] Setting key f675463ca95c42461247e2314f67d634
I20260807 08:46:56.401888 19174 log.cc:824] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:56.402819 19174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74: No bootstrap required, opened a new log
I20260807 08:46:56.405135 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:56.402438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46375 (local address 127.0.0.1:34155)
0807 08:46:56.402649 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:56.402660 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:56.402702 (+ 42us) server_negotiation.cc:407] Connection header received
0807 08:46:56.403276 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:56.403285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:56.403315 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:56.403486 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:56.404431 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:56.404435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:56.404436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:56.404440 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:56.404490 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:56.404503 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:56.404642 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:56.404696 (+ 54us) server_negotiation.cc:299] Negotiation successful
0807 08:46:56.404746 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260807 08:46:56.405085 19174 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a92ac74df42ef8193dc6ec00bfa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } }
I20260807 08:46:56.405323 19174 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:56.405376 19174 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7a92ac74df42ef8193dc6ec00bfa74, State: Initialized, Role: FOLLOWER
I20260807 08:46:56.405517 19174 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a92ac74df42ef8193dc6ec00bfa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } }
I20260807 08:46:56.405659 19174 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:56.405709 19174 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:56.405779 19174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:56.406417 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:46:56.406788 19174 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a92ac74df42ef8193dc6ec00bfa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } }
I20260807 08:46:56.406932 19174 leader_election.cc:302] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7a92ac74df42ef8193dc6ec00bfa74; no voters:
I20260807 08:46:56.407192 19174 leader_election.cc:288] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:46:56.407222 19179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:56.407364 19179 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [term 1 LEADER]: Becoming Leader. State: Replica: bc7a92ac74df42ef8193dc6ec00bfa74, State: Running, Role: LEADER
I20260807 08:46:56.407467 19179 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a92ac74df42ef8193dc6ec00bfa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } }
I20260807 08:46:56.407512 19174 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:56.407884 19179 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7a92ac74df42ef8193dc6ec00bfa74. Latest consensus state: current_term: 1 leader_uuid: "bc7a92ac74df42ef8193dc6ec00bfa74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a92ac74df42ef8193dc6ec00bfa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } } }
I20260807 08:46:56.407979 19179 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:56.408039 19180 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc7a92ac74df42ef8193dc6ec00bfa74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7a92ac74df42ef8193dc6ec00bfa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } } }
I20260807 08:46:56.408248 19180 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:56.408332 19189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:56.409088 19189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:56.410780 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:56.408719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34789 (local address 127.0.0.1:34155)
0807 08:46:56.408773 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:56.408775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:56.409398 (+ 623us) server_negotiation.cc:407] Connection header received
0807 08:46:56.409494 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:56.409495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:56.409501 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:56.409555 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:56.410470 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:56.410472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:56.410472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:56.410478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:56.410494 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:56.410499 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:56.410560 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:56.410658 (+ 98us) server_negotiation.cc:299] Negotiation successful
0807 08:46:56.410674 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 08:46:56.414220 19189 catalog_manager.cc:1347] Generated new cluster ID: 460b072793244e6fb3c10b805fcf160f
I20260807 08:46:56.414309 19189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:46:56.430773 19189 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:46:56.430997 19189 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:56.440332 19189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc7a92ac74df42ef8193dc6ec00bfa74: Generated new TSK 0
I20260807 08:46:56.440562 19189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:46:56.445745 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 19098
2026-08-07T08:46:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1508 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:46:56 dist-test-slave-9zdj krb5kdc[19206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:46:56 dist-test-slave-9zdj krb5kdc[19206](info): set up 2 sockets
Aug 07 08:46:56 dist-test-slave-9zdj krb5kdc[19206](info): commencing operation
krb5kdc: starting...
W20260807 08:46:57.579087 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:46:57 dist-test-slave-9zdj krb5kdc[19206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:46:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:46:57.735049 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:57.826381 19220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:57.826717 19220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:57.826748 19220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:57.828088 19220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:57.828154 19220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:57.828169 19220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:57.828183 19220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:57.828195 19220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:57.830030 19220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:57.830384 19220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:57.830664 19220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:46:57.836354 19226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:57.836346 19225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:57.836354 19228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:57.836915 19220 server_base.cc:1034] running on GCE node
I20260807 08:46:57.837239 19220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:57.837680 19220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:57.838897 19220 hybrid_clock.cc:648] HybridClock initialized: now 1786092417838879 us; error 35 us; skew 500 ppm
Aug 07 08:46:57 dist-test-slave-9zdj krb5kdc[19206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:57.842684 19220 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:46:57.843291 19220 webserver.cc:460] Webserver started at http://127.0.0.1:34159/ using document root <none> and password file <none>
I20260807 08:46:57.843582 19220 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:57.843636 19220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:57.843797 19220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:57.845064 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "77481f6b347b4a58855eb8c2d8c2005b"
format_stamp: "Formatted at 2026-08-07 08:46:57 on dist-test-slave-9zdj"
server_key: "9aa1934da7961ead92734475b5d46f33"
server_key_iv: "34202c2861fead460910c275ff69df35"
server_key_version: "encryptionkey@0"
I20260807 08:46:57.845546 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "77481f6b347b4a58855eb8c2d8c2005b"
format_stamp: "Formatted at 2026-08-07 08:46:57 on dist-test-slave-9zdj"
server_key: "9aa1934da7961ead92734475b5d46f33"
server_key_iv: "34202c2861fead460910c275ff69df35"
server_key_version: "encryptionkey@0"
I20260807 08:46:57.847420 19220 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260807 08:46:57.848592 19235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:57.848829 19220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:57.848986 19220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "77481f6b347b4a58855eb8c2d8c2005b"
format_stamp: "Formatted at 2026-08-07 08:46:57 on dist-test-slave-9zdj"
server_key: "9aa1934da7961ead92734475b5d46f33"
server_key_iv: "34202c2861fead460910c275ff69df35"
server_key_version: "encryptionkey@0"
I20260807 08:46:57.849086 19220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:57.873039 19220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:57.875855 19220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:57.876068 19220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:57.880913 19220 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36823
I20260807 08:46:57.881049 19293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36823 every 8 connection(s)
I20260807 08:46:57.881584 19220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:57.882503 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:57.882556 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 19220
I20260807 08:46:57.882710 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:46:57.882961 29106 external_mini_cluster.cc:1428] Setting key b08bb9678dbc3487b8596e5f9ffe4519
I20260807 08:46:57.886485 19294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b: Bootstrap starting.
I20260807 08:46:57.887280 19294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:57.887620 19294 log.cc:824] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:57.888545 19294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b: No bootstrap required, opened a new log
Aug 07 08:46:57 dist-test-slave-9zdj krb5kdc[19206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 08:46:57.890690 19294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481f6b347b4a58855eb8c2d8c2005b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260807 08:46:57.890887 19294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:57.890942 19294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77481f6b347b4a58855eb8c2d8c2005b, State: Initialized, Role: FOLLOWER
I20260807 08:46:57.891089 19294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481f6b347b4a58855eb8c2d8c2005b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260807 08:46:57.891196 19294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:57.891245 19294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:57.891301 19294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:57.891959 19294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481f6b347b4a58855eb8c2d8c2005b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260807 08:46:57.892102 19294 leader_election.cc:302] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77481f6b347b4a58855eb8c2d8c2005b; no voters:
I20260807 08:46:57.892305 19294 leader_election.cc:288] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:46:57.892388 19299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:57.892565 19299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [term 1 LEADER]: Becoming Leader. State: Replica: 77481f6b347b4a58855eb8c2d8c2005b, State: Running, Role: LEADER
I20260807 08:46:57.892680 19294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:57.892746 19299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481f6b347b4a58855eb8c2d8c2005b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260807 08:46:57.893208 19299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77481f6b347b4a58855eb8c2d8c2005b. Latest consensus state: current_term: 1 leader_uuid: "77481f6b347b4a58855eb8c2d8c2005b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481f6b347b4a58855eb8c2d8c2005b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } } }
I20260807 08:46:57.893296 19299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:57.894421 19301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77481f6b347b4a58855eb8c2d8c2005b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77481f6b347b4a58855eb8c2d8c2005b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } } }
I20260807 08:46:57.894637 19301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b [sys.catalog]: This master's current role is: LEADER
W20260807 08:46:57.895110 19317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:46:57.895174 19317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:46:57.895249 19308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:57.895438 19308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:57.895632 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:57.883569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36333 (local address 127.0.0.1:36823)
0807 08:46:57.883791 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:57.883801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:57.883848 (+ 47us) server_negotiation.cc:407] Connection header received
0807 08:46:57.884405 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:57.884415 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:57.884488 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:57.884656 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:57.885361 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:57.886799 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:57.887682 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:57.888281 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:57.888408 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:57.891113 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:57.891117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:57.891118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:57.891130 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:57.893577 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:57.894095 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:57.894099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:57.894099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:57.894221 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:57.894480 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:57.894481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:57.894482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:57.894901 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:57.895103 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:57.895235 (+ 132us) server_negotiation.cc:299] Negotiation successful
0807 08:46:57.895386 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260807 08:46:57.896732 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:46:57.898181 19308 catalog_manager.cc:1347] Generated new cluster ID: 0df5a8cfe9b543df86b13d4510646848
I20260807 08:46:57.898253 19308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:46:57.904465 19308 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:46:57.905197 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:57.898228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36895 (local address 127.0.0.1:36823)
0807 08:46:57.898269 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:57.898271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:57.898439 (+ 168us) server_negotiation.cc:407] Connection header received
0807 08:46:57.898515 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:57.898516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:57.898526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:57.898574 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:57.899141 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:57.899650 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:57.900424 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:57.900658 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:57.900743 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:57.902132 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:57.902134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:57.902134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:46:57.902142 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:57.903658 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:57.904049 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:57.904051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:57.904051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:57.904117 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:46:57.904374 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:46:57.904375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:46:57.904376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:46:57.904513 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:57.904607 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:57.904834 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 08:46:57.904926 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260807 08:46:57.905349 19308 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:57.908803 19308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77481f6b347b4a58855eb8c2d8c2005b: Generated new TSK 0
I20260807 08:46:57.909049 19308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:46:57.913594 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 19220
2026-08-07T08:46:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:46:57 dist-test-slave-9zdj krb5kdc[19330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:46:57 dist-test-slave-9zdj krb5kdc[19330](info): set up 2 sockets
Aug 07 08:46:57 dist-test-slave-9zdj krb5kdc[19330](info): commencing operation
krb5kdc: starting...
W20260807 08:46:59.067869 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:46:59 dist-test-slave-9zdj krb5kdc[19330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:46:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:46:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:46:59.226294 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:46:59.315752 19346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:46:59.316005 19346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:46:59.316032 19346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:46:59.317267 19346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:46:59.317314 19346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:46:59.317329 19346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:46:59.317342 19346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:46:59.317358 19346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:46:59.318990 19346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:46:59.319252 19346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:46:59.319448 19346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:46:59.325814 19346 server_base.cc:1034] running on GCE node
W20260807 08:46:59.325731 19351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:59.325830 19352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:46:59.325731 19355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:46:59.326525 19346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:46:59.326918 19346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:46:59.328155 19346 hybrid_clock.cc:648] HybridClock initialized: now 1786092419328138 us; error 38 us; skew 500 ppm
Aug 07 08:46:59 dist-test-slave-9zdj krb5kdc[19330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:46:59.331775 19346 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:46:59.332435 19346 webserver.cc:460] Webserver started at http://127.0.0.1:46717/ using document root <none> and password file <none>
I20260807 08:46:59.332726 19346 fs_manager.cc:359] Metadata directory not provided
I20260807 08:46:59.332804 19346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:46:59.332966 19346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:46:59.334218 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "88950b6a8b2e496492909aa8c295e285"
format_stamp: "Formatted at 2026-08-07 08:46:59 on dist-test-slave-9zdj"
server_key: "e4c0f136eb62b762fc029d2aa053bb5c"
server_key_iv: "74d01a7184a42461860d5cd64184f75c"
server_key_version: "encryptionkey@0"
I20260807 08:46:59.334614 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "88950b6a8b2e496492909aa8c295e285"
format_stamp: "Formatted at 2026-08-07 08:46:59 on dist-test-slave-9zdj"
server_key: "e4c0f136eb62b762fc029d2aa053bb5c"
server_key_iv: "74d01a7184a42461860d5cd64184f75c"
server_key_version: "encryptionkey@0"
I20260807 08:46:59.336298 19346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:46:59.337347 19362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:46:59.337783 19346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:46:59.337932 19346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "88950b6a8b2e496492909aa8c295e285"
format_stamp: "Formatted at 2026-08-07 08:46:59 on dist-test-slave-9zdj"
server_key: "e4c0f136eb62b762fc029d2aa053bb5c"
server_key_iv: "74d01a7184a42461860d5cd64184f75c"
server_key_version: "encryptionkey@0"
I20260807 08:46:59.338040 19346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:46:59.356153 19346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:46:59.358891 19346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:46:59.359104 19346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:46:59.363914 19346 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36357
I20260807 08:46:59.364017 19424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36357 every 8 connection(s)
I20260807 08:46:59.364435 19346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:46:59.365561 19426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:46:59.368336 19426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285: Bootstrap starting.
I20260807 08:46:59.368986 19426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285: Neither blocks nor log segments found. Creating new log.
I20260807 08:46:59.369289 19426 log.cc:824] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285: Log is configured to *not* fsync() on all Append() calls
I20260807 08:46:59.370426 19426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285: No bootstrap required, opened a new log
I20260807 08:46:59.372416 19426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88950b6a8b2e496492909aa8c295e285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } }
I20260807 08:46:59.372604 19426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:46:59.372632 19426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88950b6a8b2e496492909aa8c295e285, State: Initialized, Role: FOLLOWER
I20260807 08:46:59.372758 19426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88950b6a8b2e496492909aa8c295e285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } }
I20260807 08:46:59.372866 19426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:46:59.372918 19426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:46:59.373005 19426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:46:59.373859 19426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88950b6a8b2e496492909aa8c295e285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } }
I20260807 08:46:59.373976 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 19346
I20260807 08:46:59.374018 19426 leader_election.cc:302] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88950b6a8b2e496492909aa8c295e285; no voters:
I20260807 08:46:59.374104 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:46:59.374289 19426 leader_election.cc:288] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:46:59.374336 29106 external_mini_cluster.cc:1428] Setting key ceeadb1cc1489d48d628b7008a799176
I20260807 08:46:59.374457 19429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:46:59.374631 19426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:46:59.374773 19429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [term 1 LEADER]: Becoming Leader. State: Replica: 88950b6a8b2e496492909aa8c295e285, State: Running, Role: LEADER
I20260807 08:46:59.374941 19429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88950b6a8b2e496492909aa8c295e285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } }
I20260807 08:46:59.375514 19432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88950b6a8b2e496492909aa8c295e285. Latest consensus state: current_term: 1 leader_uuid: "88950b6a8b2e496492909aa8c295e285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88950b6a8b2e496492909aa8c295e285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } } }
I20260807 08:46:59.375620 19432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:59.375802 19431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88950b6a8b2e496492909aa8c295e285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88950b6a8b2e496492909aa8c295e285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } } }
I20260807 08:46:59.375893 19431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285 [sys.catalog]: This master's current role is: LEADER
I20260807 08:46:59.375938 19437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:46:59.376601 19437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:46:59.379323 19437 catalog_manager.cc:1347] Generated new cluster ID: 19b6cfbb040945b187cf8628eb30c4bc
I20260807 08:46:59.379395 19437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:46:59.382640 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:59.375171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46141 (local address 127.0.0.1:36357)
0807 08:46:59.375373 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:59.375384 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:59.375651 (+ 267us) server_negotiation.cc:407] Connection header received
0807 08:46:59.375827 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:59.375837 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:59.375910 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:59.376383 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:59.376495 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:59.378630 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:59.379906 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:59.380669 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:59.380814 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:59.381726 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:59.381732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:59.381732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:59.382020 (+ 288us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:59.382073 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:59.382093 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:59.382176 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:59.382262 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 08:46:59.382352 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260807 08:46:59.383817 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:46:59.389587 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:46:59.385400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55513 (local address 127.0.0.1:36357)
0807 08:46:59.385481 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0807 08:46:59.385484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:46:59.385695 (+ 211us) server_negotiation.cc:407] Connection header received
0807 08:46:59.385756 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:46:59.385757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:46:59.385771 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:46:59.385847 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:46:59.386001 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:59.386890 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:59.387964 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:46:59.388380 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:46:59.388471 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:46:59.389287 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:46:59.389289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:46:59.389290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:46:59.389296 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:46:59.389316 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:46:59.389323 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:46:59.389389 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 08:46:59.389447 (+ 58us) server_negotiation.cc:299] Negotiation successful
0807 08:46:59.389484 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 08:46:59.391665 19437 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:46:59.392310 19437 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:46:59.397957 19437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88950b6a8b2e496492909aa8c295e285: Generated new TSK 0
I20260807 08:46:59.398216 19437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:46:59.407790 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 19346
2026-08-07T08:46:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1494 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:46:59 dist-test-slave-9zdj krb5kdc[19459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:46:59 dist-test-slave-9zdj krb5kdc[19459](info): set up 2 sockets
Aug 07 08:46:59 dist-test-slave-9zdj krb5kdc[19459](info): commencing operation
krb5kdc: starting...
W20260807 08:47:00.574690 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:00 dist-test-slave-9zdj krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:47:00.726444 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:00.828022 19474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:00.828352 19474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:00.828400 19474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:00.829972 19474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:00.830060 19474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:00.830096 19474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:00.830127 19474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:00.830163 19474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:00.832056 19474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:00.832436 19474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:00.832723 19474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:47:00.838676 19474 server_base.cc:1034] running on GCE node
W20260807 08:47:00.838620 19480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:00.838666 19482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:00.838620 19479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:00.839227 19474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:00.839607 19474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:00.840827 19474 hybrid_clock.cc:648] HybridClock initialized: now 1786092420840798 us; error 49 us; skew 500 ppm
Aug 07 08:47:00 dist-test-slave-9zdj krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:00.844058 19474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:47:00.844664 19474 webserver.cc:460] Webserver started at http://127.0.0.1:40791/ using document root <none> and password file <none>
I20260807 08:47:00.844938 19474 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:00.844988 19474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:00.845089 19474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:00.846331 19474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "c8c0cbe505ae429789bff9d96f8e680c"
format_stamp: "Formatted at 2026-08-07 08:47:00 on dist-test-slave-9zdj"
server_key: "05abfaca61676b113077aa250c1aabf0"
server_key_iv: "a59e311bb75334afae3443342e950d22"
server_key_version: "encryptionkey@0"
I20260807 08:47:00.846668 19474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "c8c0cbe505ae429789bff9d96f8e680c"
format_stamp: "Formatted at 2026-08-07 08:47:00 on dist-test-slave-9zdj"
server_key: "05abfaca61676b113077aa250c1aabf0"
server_key_iv: "a59e311bb75334afae3443342e950d22"
server_key_version: "encryptionkey@0"
I20260807 08:47:00.848126 19474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:47:00.848994 19490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:00.849211 19474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:00.849326 19474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "c8c0cbe505ae429789bff9d96f8e680c"
format_stamp: "Formatted at 2026-08-07 08:47:00 on dist-test-slave-9zdj"
server_key: "05abfaca61676b113077aa250c1aabf0"
server_key_iv: "a59e311bb75334afae3443342e950d22"
server_key_version: "encryptionkey@0"
I20260807 08:47:00.849442 19474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:00.853374 19474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:00.856237 19474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:00.856436 19474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:00.861893 19474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44041
I20260807 08:47:00.861933 19545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44041 every 8 connection(s)
I20260807 08:47:00.862360 19474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:00.863587 19546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:00.864869 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 19474
I20260807 08:47:00.865051 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:00.865309 29106 external_mini_cluster.cc:1428] Setting key 2f81d0e04b4d413b1a5d800f263081da
I20260807 08:47:00.866734 19546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c: Bootstrap starting.
I20260807 08:47:00.867578 19546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:00.867985 19546 log.cc:824] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:00.868278 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:00.866002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51863 (local address 127.0.0.1:44041)
0807 08:47:00.866336 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:00.866346 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:00.866375 (+ 29us) server_negotiation.cc:407] Connection header received
0807 08:47:00.866814 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:00.866821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:00.866890 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:00.867044 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:00.867800 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:00.867805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:00.867805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:00.867810 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:00.867865 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:00.867876 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:00.867949 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:00.868032 (+ 83us) server_negotiation.cc:299] Negotiation successful
0807 08:47:00.868091 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260807 08:47:00.869102 19546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c: No bootstrap required, opened a new log
I20260807 08:47:00.870059 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:00.872133 19546 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0cbe505ae429789bff9d96f8e680c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260807 08:47:00.872392 19546 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:00.872454 19546 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c0cbe505ae429789bff9d96f8e680c, State: Initialized, Role: FOLLOWER
I20260807 08:47:00.872617 19546 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0cbe505ae429789bff9d96f8e680c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260807 08:47:00.872745 19546 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:00.872804 19546 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:00.872880 19546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:00.873158 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:00.871763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49671 (local address 127.0.0.1:44041)
0807 08:47:00.871813 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:00.871815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:00.871921 (+ 106us) server_negotiation.cc:407] Connection header received
0807 08:47:00.872012 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:00.872013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:00.872018 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:00.872070 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:00.872876 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:00.872878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:00.872878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:00.872885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:00.872903 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:00.872908 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:00.872972 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:00.873057 (+ 85us) server_negotiation.cc:299] Negotiation successful
0807 08:47:00.873069 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 08:47:00.873947 19546 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0cbe505ae429789bff9d96f8e680c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260807 08:47:00.874097 19546 leader_election.cc:302] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8c0cbe505ae429789bff9d96f8e680c; no voters:
I20260807 08:47:00.874409 19546 leader_election.cc:288] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:00.874491 19556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:00.874719 19556 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [term 1 LEADER]: Becoming Leader. State: Replica: c8c0cbe505ae429789bff9d96f8e680c, State: Running, Role: LEADER
I20260807 08:47:00.874809 19546 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:00.874912 19556 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0cbe505ae429789bff9d96f8e680c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260807 08:47:00.875495 19557 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8c0cbe505ae429789bff9d96f8e680c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0cbe505ae429789bff9d96f8e680c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } } }
I20260807 08:47:00.875615 19557 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:00.876121 19558 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8c0cbe505ae429789bff9d96f8e680c. Latest consensus state: current_term: 1 leader_uuid: "c8c0cbe505ae429789bff9d96f8e680c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0cbe505ae429789bff9d96f8e680c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } } }
I20260807 08:47:00.876220 19558 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:00.876437 19564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:00.877046 19564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:00.879106 19564 catalog_manager.cc:1347] Generated new cluster ID: 619cb9d741c4493cabdafb6511301aa8
I20260807 08:47:00.879165 19564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:00.896940 19564 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:00.897176 19564 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:00.902897 19564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8c0cbe505ae429789bff9d96f8e680c: Generated new TSK 0
I20260807 08:47:00.903154 19564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:00.914008 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 19474
2026-08-07T08:47:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1504 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:00 dist-test-slave-9zdj krb5kdc[19580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:00 dist-test-slave-9zdj krb5kdc[19580](info): set up 2 sockets
Aug 07 08:47:00 dist-test-slave-9zdj krb5kdc[19580](info): commencing operation
krb5kdc: starting...
W20260807 08:47:02.122548 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:02 dist-test-slave-9zdj krb5kdc[19580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:47:02.275130 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:02.363561 19593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:02.363859 19593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:02.363901 19593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:02.365208 19593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:02.365296 19593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:02.365319 19593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:02.365340 19593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:02.365367 19593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:02.367129 19593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:02.367479 19593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:02.367762 19593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:02.373672 19598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:02.373708 19599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:02.373709 19601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:02.373898 19593 server_base.cc:1034] running on GCE node
I20260807 08:47:02.374292 19593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:02.374640 19593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:02.375849 19593 hybrid_clock.cc:648] HybridClock initialized: now 1786092422375830 us; error 30 us; skew 500 ppm
Aug 07 08:47:02 dist-test-slave-9zdj krb5kdc[19580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:02.384147 19593 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:47:02.384781 19593 webserver.cc:460] Webserver started at http://127.0.0.1:39309/ using document root <none> and password file <none>
I20260807 08:47:02.385073 19593 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:02.385311 19593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:02.385529 19593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:02.386915 19593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "4f1ea9ecd72747c99541b9c6a976b525"
format_stamp: "Formatted at 2026-08-07 08:47:02 on dist-test-slave-9zdj"
server_key: "b32026dbd8328afa2f49fd25bbb57b73"
server_key_iv: "81761330ee53df04dc00196305749096"
server_key_version: "encryptionkey@0"
I20260807 08:47:02.387367 19593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "4f1ea9ecd72747c99541b9c6a976b525"
format_stamp: "Formatted at 2026-08-07 08:47:02 on dist-test-slave-9zdj"
server_key: "b32026dbd8328afa2f49fd25bbb57b73"
server_key_iv: "81761330ee53df04dc00196305749096"
server_key_version: "encryptionkey@0"
I20260807 08:47:02.389104 19593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260807 08:47:02.390458 19609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:02.390765 19593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:47:02.390924 19593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "4f1ea9ecd72747c99541b9c6a976b525"
format_stamp: "Formatted at 2026-08-07 08:47:02 on dist-test-slave-9zdj"
server_key: "b32026dbd8328afa2f49fd25bbb57b73"
server_key_iv: "81761330ee53df04dc00196305749096"
server_key_version: "encryptionkey@0"
I20260807 08:47:02.391026 19593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:02.432154 19593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:02.435642 19593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:02.435810 19593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:02.440924 19593 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43279
I20260807 08:47:02.440968 19669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43279 every 8 connection(s)
I20260807 08:47:02.441439 19593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:02.442617 19670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:02.443432 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 19593
I20260807 08:47:02.443549 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:02.443809 29106 external_mini_cluster.cc:1428] Setting key 990a0cf1f218a0d00563d70f919f5159
I20260807 08:47:02.446574 19670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525: Bootstrap starting.
I20260807 08:47:02.447428 19670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:02.447795 19670 log.cc:824] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:02.448804 19670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525: No bootstrap required, opened a new log
I20260807 08:47:02.452069 19670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1ea9ecd72747c99541b9c6a976b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43279 } }
I20260807 08:47:02.452265 19670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:02.452293 19670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f1ea9ecd72747c99541b9c6a976b525, State: Initialized, Role: FOLLOWER
I20260807 08:47:02.452443 19670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1ea9ecd72747c99541b9c6a976b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43279 } }
I20260807 08:47:02.452531 19670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:02.452558 19670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:02.452618 19670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [term 0 FOLLOWER]: Advancing to term 1
Aug 07 08:47:02 dist-test-slave-9zdj krb5kdc[19580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 08:47:02.453450 19670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1ea9ecd72747c99541b9c6a976b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43279 } }
I20260807 08:47:02.453588 19670 leader_election.cc:302] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f1ea9ecd72747c99541b9c6a976b525; no voters:
I20260807 08:47:02.453847 19670 leader_election.cc:288] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:02.454093 19677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:02.454210 19670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:02.454308 19677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [term 1 LEADER]: Becoming Leader. State: Replica: 4f1ea9ecd72747c99541b9c6a976b525, State: Running, Role: LEADER
I20260807 08:47:02.454454 19677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1ea9ecd72747c99541b9c6a976b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43279 } }
I20260807 08:47:02.454994 19676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f1ea9ecd72747c99541b9c6a976b525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1ea9ecd72747c99541b9c6a976b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43279 } } }
I20260807 08:47:02.455013 19678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f1ea9ecd72747c99541b9c6a976b525. Latest consensus state: current_term: 1 leader_uuid: "4f1ea9ecd72747c99541b9c6a976b525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f1ea9ecd72747c99541b9c6a976b525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43279 } } }
I20260807 08:47:02.455113 19676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:02.455142 19678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:02.455367 19686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:02.456111 19686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:02.458258 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:02.444492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:55535 (local address 127.0.0.1:43279)
0807 08:47:02.444752 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:02.444762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:02.444792 (+ 30us) server_negotiation.cc:407] Connection header received
0807 08:47:02.445283 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:02.445291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:02.445347 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:02.445484 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:02.446513 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:02.447880 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:02.449127 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:02.449671 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:02.449791 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:02.453361 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:02.453368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:02.453368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:02.453384 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:02.456120 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:02.456741 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:02.456745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:02.456746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:02.456853 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:02.457129 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:02.457131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:02.457131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:02.457417 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:02.457608 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:02.457889 (+ 281us) server_negotiation.cc:299] Negotiation successful
0807 08:47:02.458016 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260807 08:47:02.458267 19686 catalog_manager.cc:1347] Generated new cluster ID: 00d4a1555c1d45e0bf84a6cc02cf08cf
I20260807 08:47:02.458355 19686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:02.459517 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:02.469120 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:02.461112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:32867 (local address 127.0.0.1:43279)
0807 08:47:02.461162 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:02.461165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:02.461306 (+ 141us) server_negotiation.cc:407] Connection header received
0807 08:47:02.461380 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:02.461382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:02.461394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:02.461451 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:02.462144 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:02.463043 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:02.463889 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:02.464194 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:02.464288 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:02.465910 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:02.465914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:02.465915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:02.465924 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:02.467664 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:02.468130 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:02.468131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:02.468132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:02.468207 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:02.468478 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:02.468479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:02.468479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:02.468619 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:02.468752 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:02.468910 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 08:47:02.468989 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 08:47:02.486635 19686 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:02.487382 19686 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:02.507941 19686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f1ea9ecd72747c99541b9c6a976b525: Generated new TSK 0
I20260807 08:47:02.508224 19686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:02.514938 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 19593
W20260807 08:47:02.522068 19576 connection.cc:535] client connection to 127.0.0.1:43279 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43279): Connection reset by peer (error 104)
2026-08-07T08:47:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1600 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:02 dist-test-slave-9zdj krb5kdc[19705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:02 dist-test-slave-9zdj krb5kdc[19705](info): set up 2 sockets
Aug 07 08:47:02 dist-test-slave-9zdj krb5kdc[19705](info): commencing operation
krb5kdc: starting...
W20260807 08:47:03.670070 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:03 dist-test-slave-9zdj krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:47:03.825227 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:03.916842 19719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:03.917363 19719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:03.917459 19719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:03.918898 19719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:03.918989 19719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:03.919015 19719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:03.919037 19719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:03.919070 19719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:03.920903 19719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:03.921285 19719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:03.921582 19719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:03.927739 19728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:03.927793 19725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:03.927834 19719 server_base.cc:1034] running on GCE node
W20260807 08:47:03.927764 19726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:03.928383 19719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:03.928808 19719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:03.929993 19719 hybrid_clock.cc:648] HybridClock initialized: now 1786092423929988 us; error 46 us; skew 500 ppm
Aug 07 08:47:03 dist-test-slave-9zdj krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:03.933266 19719 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:47:03.934015 19719 webserver.cc:460] Webserver started at http://127.0.0.1:36155/ using document root <none> and password file <none>
I20260807 08:47:03.934373 19719 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:03.934463 19719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:03.934655 19719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:03.935966 19719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "0b3cb9c6a6424fb1bf1916931167b590"
format_stamp: "Formatted at 2026-08-07 08:47:03 on dist-test-slave-9zdj"
server_key: "2fb69ddfbfb56eda19d062b4829e12c2"
server_key_iv: "f1e0b0594e40f41db6a9517581a034f1"
server_key_version: "encryptionkey@0"
I20260807 08:47:03.936429 19719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "0b3cb9c6a6424fb1bf1916931167b590"
format_stamp: "Formatted at 2026-08-07 08:47:03 on dist-test-slave-9zdj"
server_key: "2fb69ddfbfb56eda19d062b4829e12c2"
server_key_iv: "f1e0b0594e40f41db6a9517581a034f1"
server_key_version: "encryptionkey@0"
I20260807 08:47:03.938259 19719 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 08:47:03.939366 19735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:03.939707 19719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:03.939854 19719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "0b3cb9c6a6424fb1bf1916931167b590"
format_stamp: "Formatted at 2026-08-07 08:47:03 on dist-test-slave-9zdj"
server_key: "2fb69ddfbfb56eda19d062b4829e12c2"
server_key_iv: "f1e0b0594e40f41db6a9517581a034f1"
server_key_version: "encryptionkey@0"
I20260807 08:47:03.939947 19719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:03.953775 19719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:03.956554 19719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:03.956851 19719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:03.962113 19719 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35837
I20260807 08:47:03.962203 19794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35837 every 8 connection(s)
I20260807 08:47:03.962770 19719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:03.963176 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 19719
I20260807 08:47:03.963322 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:03.963600 29106 external_mini_cluster.cc:1428] Setting key 059cb7f5959f44f033fa489ea8b438e8
I20260807 08:47:03.964066 19795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:03.970579 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:03.964467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:33427 (local address 127.0.0.1:35837)
0807 08:47:03.964691 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:03.964703 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:03.964738 (+ 35us) server_negotiation.cc:407] Connection header received
0807 08:47:03.965557 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:03.965569 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:03.965682 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:03.965877 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:03.966178 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:03.967263 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:03.968138 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:03.968622 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:03.968766 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:03.969591 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:03.969597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:03.969597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:03.969871 (+ 274us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:03.969931 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:03.969948 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:03.970058 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:03.970221 (+ 163us) server_negotiation.cc:299] Negotiation successful
0807 08:47:03.970258 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20260807 08:47:03.972205 19795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590: Bootstrap starting.
I20260807 08:47:03.972788 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:03.973076 19795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:03.973408 19795 log.cc:824] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:03.974339 19795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590: No bootstrap required, opened a new log
I20260807 08:47:03.976980 19795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3cb9c6a6424fb1bf1916931167b590" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } }
I20260807 08:47:03.977239 19795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:03.977303 19795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3cb9c6a6424fb1bf1916931167b590, State: Initialized, Role: FOLLOWER
I20260807 08:47:03.977466 19795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3cb9c6a6424fb1bf1916931167b590" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } }
I20260807 08:47:03.977573 19795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:03.977654 19795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:03.977705 19795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:03.978541 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:03.974439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:38329 (local address 127.0.0.1:35837)
0807 08:47:03.974478 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:03.974480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:03.974650 (+ 170us) server_negotiation.cc:407] Connection header received
0807 08:47:03.974705 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:03.974706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:03.974718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:03.974808 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:03.974901 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:03.975674 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:03.976590 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:03.976896 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:03.976980 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:03.978128 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:03.978130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:03.978130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:03.978136 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:03.978154 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:03.978159 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:03.978237 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:03.978408 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 08:47:03.978422 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260807 08:47:03.978639 19795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3cb9c6a6424fb1bf1916931167b590" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } }
I20260807 08:47:03.978767 19795 leader_election.cc:302] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b3cb9c6a6424fb1bf1916931167b590; no voters:
I20260807 08:47:03.979017 19795 leader_election.cc:288] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:03.979085 19808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:03.979346 19795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:03.979344 19808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [term 1 LEADER]: Becoming Leader. State: Replica: 0b3cb9c6a6424fb1bf1916931167b590, State: Running, Role: LEADER
I20260807 08:47:03.979545 19808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3cb9c6a6424fb1bf1916931167b590" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } }
I20260807 08:47:03.980093 19809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b3cb9c6a6424fb1bf1916931167b590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3cb9c6a6424fb1bf1916931167b590" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } } }
I20260807 08:47:03.980106 19810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b3cb9c6a6424fb1bf1916931167b590. Latest consensus state: current_term: 1 leader_uuid: "0b3cb9c6a6424fb1bf1916931167b590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3cb9c6a6424fb1bf1916931167b590" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } } }
I20260807 08:47:03.980315 19809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:03.980453 19810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:03.980760 19820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:03.981380 19820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:03.983464 19820 catalog_manager.cc:1347] Generated new cluster ID: 88543526b1f2475e8171846d1f9a383c
I20260807 08:47:03.983567 19820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:03.999169 19820 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:03.999989 19820 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:04.004031 19820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b3cb9c6a6424fb1bf1916931167b590: Generated new TSK 0
I20260807 08:47:04.004427 19820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:04.013065 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 19719
W20260807 08:47:04.022622 19703 connection.cc:535] client connection to 127.0.0.1:35837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35837): Connection reset by peer (error 104)
2026-08-07T08:47:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1500 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:04 dist-test-slave-9zdj krb5kdc[19831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:04 dist-test-slave-9zdj krb5kdc[19831](info): set up 2 sockets
Aug 07 08:47:04 dist-test-slave-9zdj krb5kdc[19831](info): commencing operation
krb5kdc: starting...
W20260807 08:47:05.177309 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:05 dist-test-slave-9zdj krb5kdc[19831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:47:05.335098 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:05.433401 19846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:05.433678 19846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:05.433709 19846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:05.434980 19846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:05.435032 19846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:05.435047 19846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:05.435060 19846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:05.435073 19846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:05.436738 19846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:05.437004 19846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:05.437217 19846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:05.443957 19852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:05.443998 19846 server_base.cc:1034] running on GCE node
W20260807 08:47:05.443957 19854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:05.444046 19851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:05.444811 19846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:05.445150 19846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:05.446322 19846 hybrid_clock.cc:648] HybridClock initialized: now 1786092425446294 us; error 42 us; skew 500 ppm
Aug 07 08:47:05 dist-test-slave-9zdj krb5kdc[19831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:05.449994 19846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:47:05.450585 19846 webserver.cc:460] Webserver started at http://127.0.0.1:34089/ using document root <none> and password file <none>
I20260807 08:47:05.450862 19846 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:05.450947 19846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:05.451109 19846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:05.452288 19846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "faeeedf811af45be8312d0e95c1edc02"
format_stamp: "Formatted at 2026-08-07 08:47:05 on dist-test-slave-9zdj"
server_key: "b92e67c621f6d34480cc2c6456a07a7f"
server_key_iv: "71e127c94a0d059e977adf3ecf879e9f"
server_key_version: "encryptionkey@0"
I20260807 08:47:05.452736 19846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "faeeedf811af45be8312d0e95c1edc02"
format_stamp: "Formatted at 2026-08-07 08:47:05 on dist-test-slave-9zdj"
server_key: "b92e67c621f6d34480cc2c6456a07a7f"
server_key_iv: "71e127c94a0d059e977adf3ecf879e9f"
server_key_version: "encryptionkey@0"
I20260807 08:47:05.454485 19846 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 08:47:05.455677 19861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:05.456126 19846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:05.456287 19846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "faeeedf811af45be8312d0e95c1edc02"
format_stamp: "Formatted at 2026-08-07 08:47:05 on dist-test-slave-9zdj"
server_key: "b92e67c621f6d34480cc2c6456a07a7f"
server_key_iv: "71e127c94a0d059e977adf3ecf879e9f"
server_key_version: "encryptionkey@0"
I20260807 08:47:05.456411 19846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:05.461202 19846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:05.464202 19846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:05.464416 19846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:05.470346 19846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34209
I20260807 08:47:05.470438 19921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34209 every 8 connection(s)
I20260807 08:47:05.470877 19846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:05.472244 19922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:05.472651 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 19846
I20260807 08:47:05.472842 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:05.473057 29106 external_mini_cluster.cc:1428] Setting key 93044dec0bdcf96eaae6064e7c8a5055
I20260807 08:47:05.475952 19922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02: Bootstrap starting.
I20260807 08:47:05.476223 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:05.473791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:32867 (local address 127.0.0.1:34209)
0807 08:47:05.474037 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:05.474050 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:05.474086 (+ 36us) server_negotiation.cc:407] Connection header received
0807 08:47:05.474574 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:05.474582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:05.474613 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:05.474752 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:05.475530 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:05.475535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:05.475535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:05.475539 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:05.475587 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:05.475600 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:05.475673 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:05.475796 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 08:47:05.475948 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":115,"threads_started":1}
I20260807 08:47:05.476807 19922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:05.477185 19922 log.cc:824] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:05.477672 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:05.478164 19922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02: No bootstrap required, opened a new log
I20260807 08:47:05.480855 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:05.479493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:34527 (local address 127.0.0.1:34209)
0807 08:47:05.479567 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:05.479569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:05.479671 (+ 102us) server_negotiation.cc:407] Connection header received
0807 08:47:05.479725 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:05.479726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:05.479732 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:05.479811 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:05.480514 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:05.480516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:05.480516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:05.480522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:05.480542 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:05.480548 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:05.480631 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:05.480714 (+ 83us) server_negotiation.cc:299] Negotiation successful
0807 08:47:05.480730 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 08:47:05.481467 19922 raft_consensus.cc:348] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeeedf811af45be8312d0e95c1edc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } }
I20260807 08:47:05.481796 19922 raft_consensus.cc:374] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:05.481866 19922 raft_consensus.cc:729] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faeeedf811af45be8312d0e95c1edc02, State: Initialized, Role: FOLLOWER
I20260807 08:47:05.482026 19922 consensus_queue.cc:260] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeeedf811af45be8312d0e95c1edc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } }
I20260807 08:47:05.482126 19922 raft_consensus.cc:388] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:05.482154 19922 raft_consensus.cc:482] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:05.482190 19922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:05.483108 19922 raft_consensus.cc:504] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeeedf811af45be8312d0e95c1edc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } }
I20260807 08:47:05.483230 19922 leader_election.cc:302] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faeeedf811af45be8312d0e95c1edc02; no voters:
I20260807 08:47:05.483448 19922 leader_election.cc:288] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:05.483544 19934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:05.483690 19934 raft_consensus.cc:686] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [term 1 LEADER]: Becoming Leader. State: Replica: faeeedf811af45be8312d0e95c1edc02, State: Running, Role: LEADER
I20260807 08:47:05.483754 19922 sys_catalog.cc:564] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:05.483862 19934 consensus_queue.cc:237] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeeedf811af45be8312d0e95c1edc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } }
I20260807 08:47:05.484442 19935 sys_catalog.cc:455] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faeeedf811af45be8312d0e95c1edc02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeeedf811af45be8312d0e95c1edc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } } }
I20260807 08:47:05.484608 19936 sys_catalog.cc:455] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faeeedf811af45be8312d0e95c1edc02. Latest consensus state: current_term: 1 leader_uuid: "faeeedf811af45be8312d0e95c1edc02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeeedf811af45be8312d0e95c1edc02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } } }
I20260807 08:47:05.484686 19935 sys_catalog.cc:458] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:05.484776 19936 sys_catalog.cc:458] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:05.485266 19945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:05.485841 19945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:05.488162 19945 catalog_manager.cc:1347] Generated new cluster ID: 0af46b26a6c4447381a31cdac7d9aa07
I20260807 08:47:05.488330 19945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:05.500011 19945 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:05.500381 19945 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:05.511510 19945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faeeedf811af45be8312d0e95c1edc02: Generated new TSK 0
I20260807 08:47:05.511795 19945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:05.521391 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 19846
2026-08-07T08:47:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:05 dist-test-slave-9zdj krb5kdc[19958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:05 dist-test-slave-9zdj krb5kdc[19958](info): set up 2 sockets
Aug 07 08:47:05 dist-test-slave-9zdj krb5kdc[19958](info): commencing operation
krb5kdc: starting...
W20260807 08:47:06.711583 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:06 dist-test-slave-9zdj krb5kdc[19958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:47:06.866094 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:06.956434 19974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:06.956717 19974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:06.956760 19974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:06.958050 19974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:06.958137 19974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:06.958159 19974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:06.958181 19974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:06.958213 19974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:06.959813 19974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:06.960135 19974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:06.960407 19974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:06.966202 19981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:06.966429 19974 server_base.cc:1034] running on GCE node
W20260807 08:47:06.966223 19980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:06.966382 19983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:06.966861 19974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:06.967244 19974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:06.968461 19974 hybrid_clock.cc:648] HybridClock initialized: now 1786092426968412 us; error 64 us; skew 500 ppm
Aug 07 08:47:06 dist-test-slave-9zdj krb5kdc[19958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:06.971794 19974 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:47:06.972340 19974 webserver.cc:460] Webserver started at http://127.0.0.1:34179/ using document root <none> and password file <none>
I20260807 08:47:06.972615 19974 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:06.972673 19974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:06.972889 19974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:06.974134 19974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "a0dd68b9572f49e9be6421d6eb8024ae"
format_stamp: "Formatted at 2026-08-07 08:47:06 on dist-test-slave-9zdj"
server_key: "5db8e288101571d7d04e0bbe1bb4bc1a"
server_key_iv: "7badd6b1b5ee842055ac1d1a99ec211c"
server_key_version: "encryptionkey@0"
I20260807 08:47:06.974495 19974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "a0dd68b9572f49e9be6421d6eb8024ae"
format_stamp: "Formatted at 2026-08-07 08:47:06 on dist-test-slave-9zdj"
server_key: "5db8e288101571d7d04e0bbe1bb4bc1a"
server_key_iv: "7badd6b1b5ee842055ac1d1a99ec211c"
server_key_version: "encryptionkey@0"
I20260807 08:47:06.976346 19974 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 08:47:06.977499 19991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:06.977880 19974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:06.978027 19974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "a0dd68b9572f49e9be6421d6eb8024ae"
format_stamp: "Formatted at 2026-08-07 08:47:06 on dist-test-slave-9zdj"
server_key: "5db8e288101571d7d04e0bbe1bb4bc1a"
server_key_iv: "7badd6b1b5ee842055ac1d1a99ec211c"
server_key_version: "encryptionkey@0"
I20260807 08:47:06.978143 19974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:06.988973 19974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:06.991807 19974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:06.992007 19974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:06.996718 19974 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33143
I20260807 08:47:06.996778 20054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33143 every 8 connection(s)
I20260807 08:47:06.997215 19974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:06.998345 20055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:07.001259 20055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae: Bootstrap starting.
I20260807 08:47:07.002182 20055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:07.002518 20055 log.cc:824] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:07.003456 20055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae: No bootstrap required, opened a new log
I20260807 08:47:07.003559 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 19974
I20260807 08:47:07.003727 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:07.003937 29106 external_mini_cluster.cc:1428] Setting key 7792c8a23a3f5bfdfa642194319e9630
I20260807 08:47:07.005931 20055 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dd68b9572f49e9be6421d6eb8024ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260807 08:47:07.006140 20055 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:07.006181 20055 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0dd68b9572f49e9be6421d6eb8024ae, State: Initialized, Role: FOLLOWER
I20260807 08:47:07.006378 20055 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dd68b9572f49e9be6421d6eb8024ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260807 08:47:07.006484 20055 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:07.006524 20055 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:07.006572 20055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:07.007874 20055 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dd68b9572f49e9be6421d6eb8024ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260807 08:47:07.007992 20055 leader_election.cc:302] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0dd68b9572f49e9be6421d6eb8024ae; no voters:
I20260807 08:47:07.008234 20055 leader_election.cc:288] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:07.008265 20060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:07.008441 20060 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [term 1 LEADER]: Becoming Leader. State: Replica: a0dd68b9572f49e9be6421d6eb8024ae, State: Running, Role: LEADER
I20260807 08:47:07.008558 20055 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:07.008639 20060 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dd68b9572f49e9be6421d6eb8024ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260807 08:47:07.009104 20060 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0dd68b9572f49e9be6421d6eb8024ae. Latest consensus state: current_term: 1 leader_uuid: "a0dd68b9572f49e9be6421d6eb8024ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dd68b9572f49e9be6421d6eb8024ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } } }
I20260807 08:47:07.009330 20062 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0dd68b9572f49e9be6421d6eb8024ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dd68b9572f49e9be6421d6eb8024ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } } }
I20260807 08:47:07.009488 20062 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:07.009790 20060 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:07.010015 20066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:07.010851 20066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:07.013038 20066 catalog_manager.cc:1347] Generated new cluster ID: 4fd7744e8bb74dbcad935e66fb8d762e
I20260807 08:47:07.013109 20066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:47:07 dist-test-slave-9zdj krb5kdc[19958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 08:47:07.019893 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:07.004545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:51863 (local address 127.0.0.1:33143)
0807 08:47:07.004790 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:07.004799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:07.004830 (+ 31us) server_negotiation.cc:407] Connection header received
0807 08:47:07.005900 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:07.005909 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:07.005977 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:07.006120 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:07.007125 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:07.008515 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:07.010143 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:07.010925 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:07.011072 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:07.014716 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:07.014721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:07.014721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:07.014736 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:07.017285 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:07.017927 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:07.017931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:07.017932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:07.018122 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:07.018425 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:07.018426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:07.018426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:07.018860 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:07.019067 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:07.019428 (+ 361us) server_negotiation.cc:299] Negotiation successful
0807 08:47:07.019570 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":107,"threads_started":1}
I20260807 08:47:07.021552 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:07.025470 20066 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:07.026131 20066 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:07.031756 20066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0dd68b9572f49e9be6421d6eb8024ae: Generated new TSK 0
I20260807 08:47:07.031991 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:07.023297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:38481 (local address 127.0.0.1:33143)
0807 08:47:07.023346 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:07.023348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:07.023551 (+ 203us) server_negotiation.cc:407] Connection header received
0807 08:47:07.023609 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:07.023610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:07.023623 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:07.023697 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:07.024492 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:07.025290 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:07.026482 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:07.026809 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:07.026854 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:07.028618 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:07.028620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:07.028620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:07.028628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:07.030385 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:07.030872 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:07.030873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:07.030874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:07.030951 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:07.031304 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:07.031306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:07.031306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:07.031467 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:07.031620 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:07.031814 (+ 194us) server_negotiation.cc:299] Negotiation successful
0807 08:47:07.031898 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 08:47:07.032130 20066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:07.040796 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 19974
W20260807 08:47:07.047999 19956 connection.cc:535] client connection to 127.0.0.1:33143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33143): Connection reset by peer (error 104)
2026-08-07T08:47:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1516 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:07 dist-test-slave-9zdj krb5kdc[20086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:07 dist-test-slave-9zdj krb5kdc[20086](info): set up 2 sockets
Aug 07 08:47:07 dist-test-slave-9zdj krb5kdc[20086](info): commencing operation
krb5kdc: starting...
W20260807 08:47:08.214145 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:08 dist-test-slave-9zdj krb5kdc[20086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:47:08.365864 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:08.454006 20100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:08.454294 20100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:08.454339 20100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:08.455618 20100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:08.455703 20100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:08.455726 20100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:08.455746 20100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:08.455775 20100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:08.457392 20100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:08.457732 20100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:08.458001 20100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:08.463559 20106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:08.463554 20108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:08.463577 20105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:08.463752 20100 server_base.cc:1034] running on GCE node
I20260807 08:47:08.464133 20100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:08.464620 20100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:08.465801 20100 hybrid_clock.cc:648] HybridClock initialized: now 1786092428465757 us; error 55 us; skew 500 ppm
Aug 07 08:47:08 dist-test-slave-9zdj krb5kdc[20086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:08.473781 20100 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:47:08.474433 20100 webserver.cc:460] Webserver started at http://127.0.0.1:43997/ using document root <none> and password file <none>
I20260807 08:47:08.474730 20100 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:08.474807 20100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:08.474970 20100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:08.476111 20100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "d07b95191e6242c88b185fdc1877d88d"
format_stamp: "Formatted at 2026-08-07 08:47:08 on dist-test-slave-9zdj"
server_key: "8fe8be71e715a61dfa7607a2fd85db93"
server_key_iv: "66f0b5495ecc83e3513c12dda2441ef4"
server_key_version: "encryptionkey@0"
I20260807 08:47:08.476516 20100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "d07b95191e6242c88b185fdc1877d88d"
format_stamp: "Formatted at 2026-08-07 08:47:08 on dist-test-slave-9zdj"
server_key: "8fe8be71e715a61dfa7607a2fd85db93"
server_key_iv: "66f0b5495ecc83e3513c12dda2441ef4"
server_key_version: "encryptionkey@0"
I20260807 08:47:08.478080 20100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:47:08.479111 20115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:08.479408 20100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:08.479526 20100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "d07b95191e6242c88b185fdc1877d88d"
format_stamp: "Formatted at 2026-08-07 08:47:08 on dist-test-slave-9zdj"
server_key: "8fe8be71e715a61dfa7607a2fd85db93"
server_key_iv: "66f0b5495ecc83e3513c12dda2441ef4"
server_key_version: "encryptionkey@0"
I20260807 08:47:08.479614 20100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:08.499521 20100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:08.502403 20100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:08.502609 20100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:08.507678 20100 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44981
I20260807 08:47:08.507738 20175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44981 every 8 connection(s)
I20260807 08:47:08.508158 20100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:08.509161 20176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:08.512056 20176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d: Bootstrap starting.
I20260807 08:47:08.512740 20176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:08.513093 20176 log.cc:824] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:08.513195 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 20100
I20260807 08:47:08.513355 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:08.513597 29106 external_mini_cluster.cc:1428] Setting key a5c2945bcd3f8c37d05c2d88d7aff1b9
I20260807 08:47:08.514104 20176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d: No bootstrap required, opened a new log
I20260807 08:47:08.516227 20176 raft_consensus.cc:348] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b95191e6242c88b185fdc1877d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } }
I20260807 08:47:08.516422 20176 raft_consensus.cc:374] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:08.516448 20176 raft_consensus.cc:729] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d07b95191e6242c88b185fdc1877d88d, State: Initialized, Role: FOLLOWER
I20260807 08:47:08.516683 20176 consensus_queue.cc:260] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b95191e6242c88b185fdc1877d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } }
I20260807 08:47:08.516811 20176 raft_consensus.cc:388] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:08.516858 20176 raft_consensus.cc:482] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:08.516953 20176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:08.517772 20176 raft_consensus.cc:504] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b95191e6242c88b185fdc1877d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } }
I20260807 08:47:08.517910 20176 leader_election.cc:302] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d07b95191e6242c88b185fdc1877d88d; no voters:
I20260807 08:47:08.518172 20176 leader_election.cc:288] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:08.518191 20181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:08.518363 20181 raft_consensus.cc:686] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [term 1 LEADER]: Becoming Leader. State: Replica: d07b95191e6242c88b185fdc1877d88d, State: Running, Role: LEADER
I20260807 08:47:08.518548 20176 sys_catalog.cc:564] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:08.518527 20181 consensus_queue.cc:237] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b95191e6242c88b185fdc1877d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } }
I20260807 08:47:08.519068 20182 sys_catalog.cc:455] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d07b95191e6242c88b185fdc1877d88d. Latest consensus state: current_term: 1 leader_uuid: "d07b95191e6242c88b185fdc1877d88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b95191e6242c88b185fdc1877d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } } }
I20260807 08:47:08.519169 20182 sys_catalog.cc:458] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:08.519117 20183 sys_catalog.cc:455] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d07b95191e6242c88b185fdc1877d88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b95191e6242c88b185fdc1877d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } } }
I20260807 08:47:08.519405 20183 sys_catalog.cc:458] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:08.519430 20188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:08.520478 20188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:08.522547 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:08.514384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:43255 (local address 127.0.0.1:44981)
0807 08:47:08.514675 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:08.514684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:08.514724 (+ 40us) server_negotiation.cc:407] Connection header received
0807 08:47:08.516231 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:08.516242 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:08.516315 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:08.516554 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:08.516895 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:08.518691 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:08.519950 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:08.520614 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:08.521099 (+ 485us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:08.521662 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:08.521667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:08.521667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:08.521916 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:08.521971 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:08.521990 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:08.522120 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:08.522267 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 08:47:08.522299 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":124,"threads_started":1}
I20260807 08:47:08.522859 20188 catalog_manager.cc:1347] Generated new cluster ID: 3cce6a8618124f1a84bb38a4817a9b5b
I20260807 08:47:08.522917 20188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:08.523800 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:08.529927 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:08.525600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:57557 (local address 127.0.0.1:44981)
0807 08:47:08.525687 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:08.525689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:08.525795 (+ 106us) server_negotiation.cc:407] Connection header received
0807 08:47:08.525870 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:08.525870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:08.525883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:08.525952 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:08.526131 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:08.526883 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:08.528136 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:08.528461 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:08.528524 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:08.529551 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:08.529552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:08.529553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:08.529558 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:08.529574 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:08.529579 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:08.529712 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:08.529820 (+ 108us) server_negotiation.cc:299] Negotiation successful
0807 08:47:08.529835 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 08:47:08.531349 20188 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:08.531862 20188 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:08.535271 20188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d07b95191e6242c88b185fdc1877d88d: Generated new TSK 0
I20260807 08:47:08.535475 20188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:08.540557 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 20100
W20260807 08:47:08.548501 20084 connection.cc:535] client connection to 127.0.0.1:44981 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44981): Connection reset by peer (error 104)
2026-08-07T08:47:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1500 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:08 dist-test-slave-9zdj krb5kdc[20209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:08 dist-test-slave-9zdj krb5kdc[20209](info): set up 2 sockets
Aug 07 08:47:08 dist-test-slave-9zdj krb5kdc[20209](info): commencing operation
krb5kdc: starting...
W20260807 08:47:09.753110 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:09 dist-test-slave-9zdj krb5kdc[20209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 08:47:09.905933 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:09.998021 20223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:09.998350 20223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:09.998400 20223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:09.999688 20223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:09.999856 20223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:09.999895 20223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:09.999930 20223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:09.999954 20223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:10.001780 20223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:10.002139 20223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:10.002424 20223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:10.008280 20228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:10.008163 20229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:10.008163 20231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:10.008661 20223 server_base.cc:1034] running on GCE node
I20260807 08:47:10.009002 20223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:10.009395 20223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:10.010609 20223 hybrid_clock.cc:648] HybridClock initialized: now 1786092430010578 us; error 69 us; skew 500 ppm
Aug 07 08:47:10 dist-test-slave-9zdj krb5kdc[20209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:10.013885 20223 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 08:47:10.014509 20223 webserver.cc:460] Webserver started at http://127.0.0.1:40493/ using document root <none> and password file <none>
I20260807 08:47:10.014819 20223 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:10.014909 20223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:10.015082 20223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:10.016440 20223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "17fb0743af8c4616b32e40aa52bf95fc"
format_stamp: "Formatted at 2026-08-07 08:47:10 on dist-test-slave-9zdj"
server_key: "9f85242b3e335d3d724ddbfb31baf35e"
server_key_iv: "0d7cde5984b87b5608e757d51acd5650"
server_key_version: "encryptionkey@0"
I20260807 08:47:10.016877 20223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "17fb0743af8c4616b32e40aa52bf95fc"
format_stamp: "Formatted at 2026-08-07 08:47:10 on dist-test-slave-9zdj"
server_key: "9f85242b3e335d3d724ddbfb31baf35e"
server_key_iv: "0d7cde5984b87b5608e757d51acd5650"
server_key_version: "encryptionkey@0"
I20260807 08:47:10.018656 20223 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 08:47:10.019663 20239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:10.020077 20223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:47:10.020193 20223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "17fb0743af8c4616b32e40aa52bf95fc"
format_stamp: "Formatted at 2026-08-07 08:47:10 on dist-test-slave-9zdj"
server_key: "9f85242b3e335d3d724ddbfb31baf35e"
server_key_iv: "0d7cde5984b87b5608e757d51acd5650"
server_key_version: "encryptionkey@0"
I20260807 08:47:10.020258 20223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:10.024248 20223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:10.027025 20223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:10.027176 20223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:10.031937 20223 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45525
I20260807 08:47:10.031992 20300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45525 every 8 connection(s)
I20260807 08:47:10.032424 20223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:10.033267 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 20223
I20260807 08:47:10.033430 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:10.033792 29106 external_mini_cluster.cc:1428] Setting key b5af0e01141977175867f1d11b90d974
I20260807 08:47:10.034111 20301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:10.036938 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:10.034448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:57049 (local address 127.0.0.1:45525)
0807 08:47:10.034644 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:10.034654 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:10.034689 (+ 35us) server_negotiation.cc:407] Connection header received
0807 08:47:10.035371 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:10.035380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:10.035412 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:10.035549 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:10.036411 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:10.036415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:10.036415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:10.036420 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:10.036465 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:10.036476 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:10.036538 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:10.036645 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 08:47:10.036710 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260807 08:47:10.038383 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:10.038887 20301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc: Bootstrap starting.
I20260807 08:47:10.039927 20301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:10.040393 20301 log.cc:824] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:10.041550 20301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc: No bootstrap required, opened a new log
I20260807 08:47:10.041781 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:10.040299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:51839 (local address 127.0.0.1:45525)
0807 08:47:10.040350 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:10.040353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:10.040480 (+ 127us) server_negotiation.cc:407] Connection header received
0807 08:47:10.040564 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:10.040565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:10.040572 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:10.040643 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:10.041459 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:10.041461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:10.041461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:10.041467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:10.041485 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:10.041491 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:10.041555 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:10.041604 (+ 49us) server_negotiation.cc:299] Negotiation successful
0807 08:47:10.041713 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 08:47:10.043902 20301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb0743af8c4616b32e40aa52bf95fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } }
I20260807 08:47:10.044061 20301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:10.044085 20301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fb0743af8c4616b32e40aa52bf95fc, State: Initialized, Role: FOLLOWER
I20260807 08:47:10.044193 20301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb0743af8c4616b32e40aa52bf95fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } }
I20260807 08:47:10.044262 20301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:10.044286 20301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:10.044317 20301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:10.044938 20301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb0743af8c4616b32e40aa52bf95fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } }
I20260807 08:47:10.045037 20301 leader_election.cc:302] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17fb0743af8c4616b32e40aa52bf95fc; no voters:
I20260807 08:47:10.045220 20301 leader_election.cc:288] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:10.045320 20313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:10.045503 20313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [term 1 LEADER]: Becoming Leader. State: Replica: 17fb0743af8c4616b32e40aa52bf95fc, State: Running, Role: LEADER
I20260807 08:47:10.045509 20301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:10.045651 20313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb0743af8c4616b32e40aa52bf95fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } }
I20260807 08:47:10.046134 20314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17fb0743af8c4616b32e40aa52bf95fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb0743af8c4616b32e40aa52bf95fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } } }
I20260807 08:47:10.046172 20316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17fb0743af8c4616b32e40aa52bf95fc. Latest consensus state: current_term: 1 leader_uuid: "17fb0743af8c4616b32e40aa52bf95fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb0743af8c4616b32e40aa52bf95fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } } }
I20260807 08:47:10.046310 20314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:10.046334 20316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc [sys.catalog]: This master's current role is: LEADER
W20260807 08:47:10.047854 20331 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:47:10.047909 20331 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:47:10.047981 20324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:10.048216 20324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:10.050071 20324 catalog_manager.cc:1347] Generated new cluster ID: 9d4abbd710cc467abaab9767bc03abd7
I20260807 08:47:10.050165 20324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:10.061651 20324 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:10.061926 20324 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:10.070695 20324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17fb0743af8c4616b32e40aa52bf95fc: Generated new TSK 0
I20260807 08:47:10.070993 20324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:10.078532 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 20223
2026-08-07T08:47:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1538 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:10 dist-test-slave-9zdj krb5kdc[20336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:10 dist-test-slave-9zdj krb5kdc[20336](info): set up 2 sockets
Aug 07 08:47:10 dist-test-slave-9zdj krb5kdc[20336](info): commencing operation
krb5kdc: starting...
W20260807 08:47:11.186636 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:11 dist-test-slave-9zdj krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:47:11.346038 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:46045
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:33607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:46045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:11.438633 20349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:11.438966 20349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:11.439011 20349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:11.440301 20349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:11.440392 20349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:11.440416 20349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:11.440438 20349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:11.440471 20349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:11.442303 20349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:33607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:46045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:11.442642 20349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:11.442962 20349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:11.448884 20355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:11.448866 20358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:11.448900 20349 server_base.cc:1034] running on GCE node
W20260807 08:47:11.448868 20354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:11.449443 20349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:11.449874 20349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:11.451045 20349 hybrid_clock.cc:648] HybridClock initialized: now 1786092431451027 us; error 32 us; skew 500 ppm
Aug 07 08:47:11 dist-test-slave-9zdj krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092431, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:11.454110 20349 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:47:11.454640 20349 webserver.cc:460] Webserver started at http://127.28.108.190:43661/ using document root <none> and password file <none>
I20260807 08:47:11.454852 20349 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:11.454897 20349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:11.454998 20349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:11.456163 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "5ad033c2b1714704bc9413a7d9e22d36"
format_stamp: "Formatted at 2026-08-07 08:47:11 on dist-test-slave-9zdj"
server_key: "26f0f9f71f47ab4ec3cabe9707775781"
server_key_iv: "a45ddddd888043e3e6b7d3fdb71973cd"
server_key_version: "encryptionkey@0"
I20260807 08:47:11.456600 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "5ad033c2b1714704bc9413a7d9e22d36"
format_stamp: "Formatted at 2026-08-07 08:47:11 on dist-test-slave-9zdj"
server_key: "26f0f9f71f47ab4ec3cabe9707775781"
server_key_iv: "a45ddddd888043e3e6b7d3fdb71973cd"
server_key_version: "encryptionkey@0"
I20260807 08:47:11.458173 20349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 08:47:11.459131 20365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:11.459434 20349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:11.459558 20349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "5ad033c2b1714704bc9413a7d9e22d36"
format_stamp: "Formatted at 2026-08-07 08:47:11 on dist-test-slave-9zdj"
server_key: "26f0f9f71f47ab4ec3cabe9707775781"
server_key_iv: "a45ddddd888043e3e6b7d3fdb71973cd"
server_key_version: "encryptionkey@0"
I20260807 08:47:11.459633 20349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:11.477030 20349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:11.479911 20349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:11.480064 20349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:11.485045 20349 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:46045
I20260807 08:47:11.485088 20420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:46045 every 8 connection(s)
I20260807 08:47:11.485579 20349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:11.486792 20421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:11.489503 20421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36: Bootstrap starting.
I20260807 08:47:11.490231 20421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:11.490573 20421 log.cc:824] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:11.491542 20421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36: No bootstrap required, opened a new log
I20260807 08:47:11.493391 20421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad033c2b1714704bc9413a7d9e22d36" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46045 } }
I20260807 08:47:11.493579 20421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:11.493713 20421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad033c2b1714704bc9413a7d9e22d36, State: Initialized, Role: FOLLOWER
I20260807 08:47:11.493943 20421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad033c2b1714704bc9413a7d9e22d36" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46045 } }
I20260807 08:47:11.494042 20421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:11.494019 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 20349
I20260807 08:47:11.494089 20421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:11.494149 20421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:11.494161 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:11.494354 29106 external_mini_cluster.cc:1428] Setting key 0cdad3dd356d8164e9e094bd2d5d7dab
I20260807 08:47:11.495628 20421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad033c2b1714704bc9413a7d9e22d36" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46045 } }
I20260807 08:47:11.495762 20421 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad033c2b1714704bc9413a7d9e22d36; no voters:
I20260807 08:47:11.496021 20421 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:11.496045 20426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:11.496168 20426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [term 1 LEADER]: Becoming Leader. State: Replica: 5ad033c2b1714704bc9413a7d9e22d36, State: Running, Role: LEADER
I20260807 08:47:11.496346 20421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:11.496340 20426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad033c2b1714704bc9413a7d9e22d36" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46045 } }
I20260807 08:47:11.496920 20428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad033c2b1714704bc9413a7d9e22d36. Latest consensus state: current_term: 1 leader_uuid: "5ad033c2b1714704bc9413a7d9e22d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad033c2b1714704bc9413a7d9e22d36" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46045 } } }
I20260807 08:47:11.497028 20428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:11.497274 20427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ad033c2b1714704bc9413a7d9e22d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad033c2b1714704bc9413a7d9e22d36" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46045 } } }
I20260807 08:47:11.497790 20436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:11.498380 20436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:11.499064 20427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:11.501238 20436 catalog_manager.cc:1347] Generated new cluster ID: f43a532f0eaf4741959e1830546a04aa
I20260807 08:47:11.501297 20436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 08:47:11 dist-test-slave-9zdj krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:47:11.508442 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:11.495001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38163 (local address 127.28.108.190:46045)
0807 08:47:11.495342 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:11.495353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:11.495394 (+ 41us) server_negotiation.cc:407] Connection header received
0807 08:47:11.495522 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:11.495530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:11.495591 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:11.495756 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:11.497015 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:11.498705 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:11.499874 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:11.500537 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:11.500668 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:11.504040 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:11.504046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:11.504046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:11.504062 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:11.506331 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:11.506902 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:11.506907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:11.506907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:11.507051 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:11.507355 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:11.507356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:11.507357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:11.507695 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:11.507938 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:11.508058 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 08:47:11.508189 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":198,"threads_started":1}
I20260807 08:47:11.509788 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:11.518973 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:11.511427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35483 (local address 127.28.108.190:46045)
0807 08:47:11.511476 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:11.511478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:11.511627 (+ 149us) server_negotiation.cc:407] Connection header received
0807 08:47:11.511717 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:11.511718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:11.511729 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:11.511785 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:11.512485 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:11.513093 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:11.514006 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:11.514273 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:11.514373 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:11.515830 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:11.515832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:11.515832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:11.515841 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:11.517480 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:11.517957 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:11.517958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:11.517958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:11.518029 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:11.518291 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:11.518292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:11.518293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:11.518431 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:11.518581 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:11.518748 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 08:47:11.518833 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 08:47:11.527045 20436 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:11.527750 20436 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:11.530745 20436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ad033c2b1714704bc9413a7d9e22d36: Generated new TSK 0
I20260807 08:47:11.530997 20436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:11.534134 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 20349
W20260807 08:47:11.542312 20334 connection.cc:535] client connection to 127.28.108.190:46045 recv error: Network error: failed to read from TLS socket (remote: 127.28.108.190:46045): Connection reset by peer (error 104)
2026-08-07T08:47:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:11 dist-test-slave-9zdj krb5kdc[20454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:11 dist-test-slave-9zdj krb5kdc[20454](info): set up 2 sockets
Aug 07 08:47:11 dist-test-slave-9zdj krb5kdc[20454](info): commencing operation
krb5kdc: starting...
W20260807 08:47:12.675590 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:12 dist-test-slave-9zdj krb5kdc[20454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:47:12.822450 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:39023
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:35383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:39023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:12.913559 20467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:12.913844 20467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:12.913872 20467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:12.915132 20467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:12.915179 20467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:12.915194 20467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:12.915207 20467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:12.915220 20467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:12.916811 20467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:35383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:39023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:39023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:12.917048 20467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:12.917258 20467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:12.922904 20473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:12.922941 20476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:12.922922 20474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:12.923223 20467 server_base.cc:1034] running on GCE node
I20260807 08:47:12.923612 20467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:12.923961 20467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:12.925163 20467 hybrid_clock.cc:648] HybridClock initialized: now 1786092432925114 us; error 87 us; skew 500 ppm
Aug 07 08:47:12 dist-test-slave-9zdj krb5kdc[20454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092432, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:12.928179 20467 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:47:12.928717 20467 webserver.cc:460] Webserver started at http://127.28.108.190:33575/ using document root <none> and password file <none>
I20260807 08:47:12.928993 20467 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:12.929071 20467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:12.929237 20467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:12.930497 20467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "37bc59cecb4e4a8d80b0252a37d86d16"
format_stamp: "Formatted at 2026-08-07 08:47:12 on dist-test-slave-9zdj"
server_key: "3baaaef53900b5c7665d820b4f0d092e"
server_key_iv: "85e35bbeaaffa6f44d4c74ac0117b4e8"
server_key_version: "encryptionkey@0"
I20260807 08:47:12.930915 20467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "37bc59cecb4e4a8d80b0252a37d86d16"
format_stamp: "Formatted at 2026-08-07 08:47:12 on dist-test-slave-9zdj"
server_key: "3baaaef53900b5c7665d820b4f0d092e"
server_key_iv: "85e35bbeaaffa6f44d4c74ac0117b4e8"
server_key_version: "encryptionkey@0"
I20260807 08:47:12.932575 20467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 08:47:12.933595 20484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:12.934023 20467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:12.934146 20467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "37bc59cecb4e4a8d80b0252a37d86d16"
format_stamp: "Formatted at 2026-08-07 08:47:12 on dist-test-slave-9zdj"
server_key: "3baaaef53900b5c7665d820b4f0d092e"
server_key_iv: "85e35bbeaaffa6f44d4c74ac0117b4e8"
server_key_version: "encryptionkey@0"
I20260807 08:47:12.934358 20467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:12.947921 20467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:12.950578 20467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:12.950778 20467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:12.955401 20467 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:39023
I20260807 08:47:12.955444 20542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:39023 every 8 connection(s)
I20260807 08:47:12.955847 20467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:12.956931 20543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:12.959529 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 20467
I20260807 08:47:12.959650 20543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16: Bootstrap starting.
I20260807 08:47:12.959686 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:12.959895 29106 external_mini_cluster.cc:1428] Setting key 118084df132a9fed4c77a82165272304
I20260807 08:47:12.960520 20543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:12.960908 20543 log.cc:824] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:12.961833 20543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16: No bootstrap required, opened a new log
I20260807 08:47:12.964346 20543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc59cecb4e4a8d80b0252a37d86d16" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39023 } }
I20260807 08:47:12.964545 20543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:12.964581 20543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bc59cecb4e4a8d80b0252a37d86d16, State: Initialized, Role: FOLLOWER
I20260807 08:47:12.964709 20543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc59cecb4e4a8d80b0252a37d86d16" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39023 } }
I20260807 08:47:12.964776 20543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:12.964805 20543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:12.964843 20543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:12.965735 20543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc59cecb4e4a8d80b0252a37d86d16" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39023 } }
I20260807 08:47:12.965857 20543 leader_election.cc:302] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37bc59cecb4e4a8d80b0252a37d86d16; no voters:
I20260807 08:47:12.966132 20543 leader_election.cc:288] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:12.966151 20549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:12.966408 20549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [term 1 LEADER]: Becoming Leader. State: Replica: 37bc59cecb4e4a8d80b0252a37d86d16, State: Running, Role: LEADER
I20260807 08:47:12.966490 20543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:12.966588 20549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc59cecb4e4a8d80b0252a37d86d16" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39023 } }
I20260807 08:47:12.967128 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:12.960543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36699 (local address 127.28.108.190:39023)
0807 08:47:12.960783 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:12.960793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:12.960833 (+ 40us) server_negotiation.cc:407] Connection header received
0807 08:47:12.961431 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:12.961441 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:12.961512 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:12.961713 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:12.961998 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:12.963411 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:12.964322 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:12.964966 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:12.965088 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:12.965999 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:12.966003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:12.966004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:12.966340 (+ 336us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:12.966395 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:12.966413 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:12.966782 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:12.966830 (+ 48us) server_negotiation.cc:299] Negotiation successful
0807 08:47:12.966860 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":105,"threads_started":1}
I20260807 08:47:12.968737 20550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37bc59cecb4e4a8d80b0252a37d86d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc59cecb4e4a8d80b0252a37d86d16" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39023 } } }
I20260807 08:47:12.968904 20550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [sys.catalog]: This master's current role is: LEADER
W20260807 08:47:12.968910 20564 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:47:12.968999 20564 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:47:12.968987 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:12.969156 20568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:12.969228 20551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37bc59cecb4e4a8d80b0252a37d86d16. Latest consensus state: current_term: 1 leader_uuid: "37bc59cecb4e4a8d80b0252a37d86d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bc59cecb4e4a8d80b0252a37d86d16" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 39023 } } }
I20260807 08:47:12.969300 20551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:12.969403 20568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:12.971202 20568 catalog_manager.cc:1347] Generated new cluster ID: b7bcf7108e8449ecac134f6df6bbadf7
I20260807 08:47:12.971268 20568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:12.974105 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:12.970701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32797 (local address 127.28.108.190:39023)
0807 08:47:12.970748 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:12.970751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:12.970956 (+ 205us) server_negotiation.cc:407] Connection header received
0807 08:47:12.971027 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:12.971028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:12.971050 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:12.971101 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:12.971361 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:12.971880 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:12.972626 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:12.972866 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:12.972948 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:12.973787 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:12.973789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:12.973789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:12.973794 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:12.973810 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:12.973815 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:12.973848 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:12.973987 (+ 139us) server_negotiation.cc:299] Negotiation successful
0807 08:47:12.973998 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:47:12.994252 20568 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:12.994926 20568 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:12.997654 20568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37bc59cecb4e4a8d80b0252a37d86d16: Generated new TSK 0
I20260807 08:47:12.997888 20568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:13.005924 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 20467
W20260807 08:47:13.014366 20452 connection.cc:535] client connection to 127.28.108.190:39023 recv error: Network error: failed to read from TLS socket (remote: 127.28.108.190:39023): Connection reset by peer (error 104)
2026-08-07T08:47:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1472 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:13 dist-test-slave-9zdj krb5kdc[20578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:13 dist-test-slave-9zdj krb5kdc[20578](info): set up 2 sockets
Aug 07 08:47:13 dist-test-slave-9zdj krb5kdc[20578](info): commencing operation
krb5kdc: starting...
W20260807 08:47:14.158064 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:14 dist-test-slave-9zdj krb5kdc[20578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:47:14.310276 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:46677
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:46677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:14.398986 20593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:14.399243 20593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:14.399271 20593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:14.400508 20593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:14.400553 20593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:14.400568 20593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:14.400583 20593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:14.400594 20593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:14.402187 20593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:46677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:46677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:14.402448 20593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:14.402643 20593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:14.408658 20602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:14.408718 20600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:14.408850 20599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:14.409739 20593 server_base.cc:1034] running on GCE node
I20260807 08:47:14.410025 20593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:14.410375 20593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:14.411602 20593 hybrid_clock.cc:648] HybridClock initialized: now 1786092434411572 us; error 41 us; skew 500 ppm
Aug 07 08:47:14 dist-test-slave-9zdj krb5kdc[20578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092434, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:14.419082 20593 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:47:14.419634 20593 webserver.cc:460] Webserver started at http://127.28.108.190:39207/ using document root <none> and password file <none>
I20260807 08:47:14.419903 20593 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:14.419977 20593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:14.420141 20593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:14.421368 20593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "5f5f2436db074dc3af455ad81a6d68af"
format_stamp: "Formatted at 2026-08-07 08:47:14 on dist-test-slave-9zdj"
server_key: "b6879ab03e96875283e0ee7e38fadeac"
server_key_iv: "e9c779be85ad2c1c09a92e9a559c2527"
server_key_version: "encryptionkey@0"
I20260807 08:47:14.421821 20593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "5f5f2436db074dc3af455ad81a6d68af"
format_stamp: "Formatted at 2026-08-07 08:47:14 on dist-test-slave-9zdj"
server_key: "b6879ab03e96875283e0ee7e38fadeac"
server_key_iv: "e9c779be85ad2c1c09a92e9a559c2527"
server_key_version: "encryptionkey@0"
I20260807 08:47:14.423414 20593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:47:14.424510 20610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:14.424782 20593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:47:14.424904 20593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "5f5f2436db074dc3af455ad81a6d68af"
format_stamp: "Formatted at 2026-08-07 08:47:14 on dist-test-slave-9zdj"
server_key: "b6879ab03e96875283e0ee7e38fadeac"
server_key_iv: "e9c779be85ad2c1c09a92e9a559c2527"
server_key_version: "encryptionkey@0"
I20260807 08:47:14.424991 20593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:14.428951 20593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:14.431512 20593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:14.431759 20593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:14.436563 20593 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:46677
I20260807 08:47:14.436616 20665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:46677 every 8 connection(s)
I20260807 08:47:14.437049 20593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:14.437606 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 20593
I20260807 08:47:14.437824 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:14.438071 29106 external_mini_cluster.cc:1428] Setting key 9cadb09a14bcad78a9cac45412d0f486
I20260807 08:47:14.438602 20666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:14.441393 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:14.438870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55465 (local address 127.28.108.190:46677)
0807 08:47:14.439092 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:14.439102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:14.439131 (+ 29us) server_negotiation.cc:407] Connection header received
0807 08:47:14.439863 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:14.439872 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:14.439912 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:14.440096 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:14.440902 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:14.440910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:14.440911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:14.440915 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:14.440957 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:14.440968 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:14.441056 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:14.441133 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 08:47:14.441183 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
I20260807 08:47:14.442292 20666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af: Bootstrap starting.
I20260807 08:47:14.442628 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:14.443099 20666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:14.443413 20666 log.cc:824] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:14.444324 20666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af: No bootstrap required, opened a new log
I20260807 08:47:14.445788 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:14.444395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56331 (local address 127.28.108.190:46677)
0807 08:47:14.444464 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:14.444466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:14.444586 (+ 120us) server_negotiation.cc:407] Connection header received
0807 08:47:14.444707 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:14.444708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:14.444713 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:14.444783 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:14.445470 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:14.445471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:14.445472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:14.445477 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:14.445493 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:14.445500 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:14.445575 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:14.445695 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 08:47:14.445710 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 08:47:14.446864 20666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5f2436db074dc3af455ad81a6d68af" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46677 } }
I20260807 08:47:14.447048 20666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:14.447099 20666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5f2436db074dc3af455ad81a6d68af, State: Initialized, Role: FOLLOWER
I20260807 08:47:14.447283 20666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5f2436db074dc3af455ad81a6d68af" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46677 } }
I20260807 08:47:14.447395 20666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:14.447444 20666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:14.447499 20666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:14.448146 20666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5f2436db074dc3af455ad81a6d68af" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46677 } }
I20260807 08:47:14.448283 20666 leader_election.cc:302] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f5f2436db074dc3af455ad81a6d68af; no voters:
I20260807 08:47:14.448518 20666 leader_election.cc:288] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:14.448561 20678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:14.448700 20678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [term 1 LEADER]: Becoming Leader. State: Replica: 5f5f2436db074dc3af455ad81a6d68af, State: Running, Role: LEADER
I20260807 08:47:14.448786 20678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5f2436db074dc3af455ad81a6d68af" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46677 } }
I20260807 08:47:14.448829 20666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:14.449193 20679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f5f2436db074dc3af455ad81a6d68af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5f2436db074dc3af455ad81a6d68af" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46677 } } }
I20260807 08:47:14.449401 20679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:14.449203 20680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f5f2436db074dc3af455ad81a6d68af. Latest consensus state: current_term: 1 leader_uuid: "5f5f2436db074dc3af455ad81a6d68af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5f2436db074dc3af455ad81a6d68af" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 46677 } } }
I20260807 08:47:14.449725 20680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:14.449951 20689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:14.450620 20689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:14.452479 20689 catalog_manager.cc:1347] Generated new cluster ID: eaa92c02e8ab4992806170840a3a5635
I20260807 08:47:14.452564 20689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:14.475453 20689 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:14.475741 20689 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:14.479617 20689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f5f2436db074dc3af455ad81a6d68af: Generated new TSK 0
I20260807 08:47:14.479955 20689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:14.486474 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 20593
2026-08-07T08:47:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1479 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:14 dist-test-slave-9zdj krb5kdc[20700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:14 dist-test-slave-9zdj krb5kdc[20700](info): set up 2 sockets
Aug 07 08:47:14 dist-test-slave-9zdj krb5kdc[20700](info): commencing operation
krb5kdc: starting...
W20260807 08:47:15.647761 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:15 dist-test-slave-9zdj krb5kdc[20700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:47:15.799049 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34863
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:34863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:15.890589 20717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:15.890875 20717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:15.890903 20717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:15.892167 20717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:15.892225 20717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:15.892241 20717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:15.892253 20717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:15.892266 20717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:15.893939 20717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:34863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:34863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:15.894196 20717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:15.894419 20717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:15.900336 20723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:15.900336 20722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:15.900374 20717 server_base.cc:1034] running on GCE node
W20260807 08:47:15.900341 20725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:15.900945 20717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:15.901258 20717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:15.902462 20717 hybrid_clock.cc:648] HybridClock initialized: now 1786092435902442 us; error 33 us; skew 500 ppm
Aug 07 08:47:15 dist-test-slave-9zdj krb5kdc[20700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092435, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:15.905818 20717 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:47:15.906342 20717 webserver.cc:460] Webserver started at http://127.28.108.190:33631/ using document root <none> and password file <none>
I20260807 08:47:15.906668 20717 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:15.906726 20717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:15.906934 20717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:15.908317 20717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "e68b34bd6f5545c0877a811539ec98db"
format_stamp: "Formatted at 2026-08-07 08:47:15 on dist-test-slave-9zdj"
server_key: "d69be19b241fc2503cce35571451ada3"
server_key_iv: "745e0142902bc06be31bd34b57b13aa9"
server_key_version: "encryptionkey@0"
I20260807 08:47:15.908809 20717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "e68b34bd6f5545c0877a811539ec98db"
format_stamp: "Formatted at 2026-08-07 08:47:15 on dist-test-slave-9zdj"
server_key: "d69be19b241fc2503cce35571451ada3"
server_key_iv: "745e0142902bc06be31bd34b57b13aa9"
server_key_version: "encryptionkey@0"
I20260807 08:47:15.910672 20717 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 08:47:15.911695 20733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:15.912015 20717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:15.912143 20717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "e68b34bd6f5545c0877a811539ec98db"
format_stamp: "Formatted at 2026-08-07 08:47:15 on dist-test-slave-9zdj"
server_key: "d69be19b241fc2503cce35571451ada3"
server_key_iv: "745e0142902bc06be31bd34b57b13aa9"
server_key_version: "encryptionkey@0"
I20260807 08:47:15.912216 20717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:15.944999 20717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:15.947839 20717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:15.947992 20717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:15.953153 20717 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:34863
I20260807 08:47:15.953189 20787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:34863 every 8 connection(s)
I20260807 08:47:15.953609 20717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:15.954649 20788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:15.956789 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 20717
I20260807 08:47:15.956905 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:15.957111 29106 external_mini_cluster.cc:1428] Setting key fcb1cbb10e35e87a16e41f7d3e7b8789
I20260807 08:47:15.957485 20788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db: Bootstrap starting.
I20260807 08:47:15.958568 20788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:15.958942 20788 log.cc:824] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:15.959753 20788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db: No bootstrap required, opened a new log
I20260807 08:47:15.962289 20788 raft_consensus.cc:348] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68b34bd6f5545c0877a811539ec98db" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34863 } }
I20260807 08:47:15.962499 20788 raft_consensus.cc:374] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:15.962536 20788 raft_consensus.cc:729] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68b34bd6f5545c0877a811539ec98db, State: Initialized, Role: FOLLOWER
I20260807 08:47:15.962666 20788 consensus_queue.cc:260] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68b34bd6f5545c0877a811539ec98db" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34863 } }
I20260807 08:47:15.962743 20788 raft_consensus.cc:388] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:15.962778 20788 raft_consensus.cc:482] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:15.962843 20788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:15.963617 20788 raft_consensus.cc:504] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68b34bd6f5545c0877a811539ec98db" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34863 } }
I20260807 08:47:15.963755 20788 leader_election.cc:302] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e68b34bd6f5545c0877a811539ec98db; no voters:
I20260807 08:47:15.964020 20788 leader_election.cc:288] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:15.964095 20793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:15.964349 20788 sys_catalog.cc:564] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:15.964522 20793 raft_consensus.cc:686] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [term 1 LEADER]: Becoming Leader. State: Replica: e68b34bd6f5545c0877a811539ec98db, State: Running, Role: LEADER
I20260807 08:47:15.964653 20793 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68b34bd6f5545c0877a811539ec98db" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34863 } }
I20260807 08:47:15.965334 20794 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e68b34bd6f5545c0877a811539ec98db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68b34bd6f5545c0877a811539ec98db" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34863 } } }
I20260807 08:47:15.965363 20795 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68b34bd6f5545c0877a811539ec98db. Latest consensus state: current_term: 1 leader_uuid: "e68b34bd6f5545c0877a811539ec98db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68b34bd6f5545c0877a811539ec98db" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 34863 } } }
I20260807 08:47:15.965435 20794 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:15.965443 20795 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db [sys.catalog]: This master's current role is: LEADER
Aug 07 08:47:15 dist-test-slave-9zdj krb5kdc[20700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
W20260807 08:47:15.966897 20809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:47:15.966960 20809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:47:15.967319 20802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:15.967532 20802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:15.969923 20802 catalog_manager.cc:1347] Generated new cluster ID: a97342f8938443caadac0184957e0234
I20260807 08:47:15.969995 20802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:15.970860 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:15.957747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41313 (local address 127.28.108.190:34863)
0807 08:47:15.957958 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:15.957968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:15.958003 (+ 35us) server_negotiation.cc:407] Connection header received
0807 08:47:15.958664 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:15.958675 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:15.958753 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:15.958929 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:15.959712 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:15.961140 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:15.962137 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:15.962801 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:15.962954 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:15.966708 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:15.966712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:15.966713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:15.966724 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:15.968913 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:15.969430 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:15.969436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:15.969441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:15.969537 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:15.969814 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:15.969815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:15.969816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:15.970171 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:15.970378 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:15.970467 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 08:47:15.970619 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":115,"threads_started":1}
I20260807 08:47:15.972025 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:15.985733 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:15.973992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58059 (local address 127.28.108.190:34863)
0807 08:47:15.974035 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:15.974037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:15.977952 (+ 3915us) server_negotiation.cc:407] Connection header received
0807 08:47:15.978052 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:15.978053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:15.978067 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:15.978126 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:15.978934 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:15.979517 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:15.980373 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:15.980635 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:15.980734 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:15.982297 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:15.982299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:15.982299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:15.982309 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:15.984125 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:15.984639 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:15.984640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:15.984641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:15.984761 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:15.984997 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:15.984998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:15.984998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:15.985178 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:15.985299 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:15.985452 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 08:47:15.985555 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 08:47:16.034053 20802 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:16.034881 20802 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:16.039522 20802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e68b34bd6f5545c0877a811539ec98db: Generated new TSK 0
I20260807 08:47:16.039834 20802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:16.049896 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 20717
2026-08-07T08:47:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1564 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:16 dist-test-slave-9zdj krb5kdc[20822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:16 dist-test-slave-9zdj krb5kdc[20822](info): set up 2 sockets
Aug 07 08:47:16 dist-test-slave-9zdj krb5kdc[20822](info): commencing operation
krb5kdc: starting...
W20260807 08:47:17.200793 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:17 dist-test-slave-9zdj krb5kdc[20822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:47:17.366935 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:40649
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:40649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:17.464974 20836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:17.465299 20836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:17.465330 20836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:17.467005 20836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:17.467100 20836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:17.467115 20836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:17.467129 20836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:17.467142 20836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:17.469251 20836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:40649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:40649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:17.469596 20836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:17.469944 20836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:17.476132 20842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:17.476122 20845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:17.476142 20841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:17.476478 20836 server_base.cc:1034] running on GCE node
I20260807 08:47:17.476902 20836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:17.477317 20836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:17.478561 20836 hybrid_clock.cc:648] HybridClock initialized: now 1786092437478507 us; error 80 us; skew 500 ppm
Aug 07 08:47:17 dist-test-slave-9zdj krb5kdc[20822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092437, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:17.481986 20836 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:47:17.482606 20836 webserver.cc:460] Webserver started at http://127.28.108.190:42919/ using document root <none> and password file <none>
I20260807 08:47:17.482951 20836 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:17.483045 20836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:17.483234 20836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:17.484716 20836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "4cd2e0ea29d14764b8fbf37feb0c6874"
format_stamp: "Formatted at 2026-08-07 08:47:17 on dist-test-slave-9zdj"
server_key: "833a723c33d252b55f374ad1d4c90988"
server_key_iv: "35f23ec70e29f9b1d720ec23bc84028d"
server_key_version: "encryptionkey@0"
I20260807 08:47:17.485199 20836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "4cd2e0ea29d14764b8fbf37feb0c6874"
format_stamp: "Formatted at 2026-08-07 08:47:17 on dist-test-slave-9zdj"
server_key: "833a723c33d252b55f374ad1d4c90988"
server_key_iv: "35f23ec70e29f9b1d720ec23bc84028d"
server_key_version: "encryptionkey@0"
I20260807 08:47:17.487035 20836 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 08:47:17.488188 20852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:17.488593 20836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:17.488808 20836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "4cd2e0ea29d14764b8fbf37feb0c6874"
format_stamp: "Formatted at 2026-08-07 08:47:17 on dist-test-slave-9zdj"
server_key: "833a723c33d252b55f374ad1d4c90988"
server_key_iv: "35f23ec70e29f9b1d720ec23bc84028d"
server_key_version: "encryptionkey@0"
I20260807 08:47:17.488909 20836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:17.499761 20836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:17.502676 20836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:17.502887 20836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:17.507750 20836 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:40649
I20260807 08:47:17.507779 20910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:40649 every 8 connection(s)
I20260807 08:47:17.508283 20836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:17.509516 20911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:17.512250 20911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874: Bootstrap starting.
I20260807 08:47:17.512993 20911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:17.513353 20911 log.cc:824] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:17.514436 20911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874: No bootstrap required, opened a new log
I20260807 08:47:17.515080 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 20836
I20260807 08:47:17.515321 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:17.515528 29106 external_mini_cluster.cc:1428] Setting key a910581619f8789f751d60fbfee323a2
I20260807 08:47:17.517643 20911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd2e0ea29d14764b8fbf37feb0c6874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40649 } }
I20260807 08:47:17.517879 20911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:17.517923 20911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd2e0ea29d14764b8fbf37feb0c6874, State: Initialized, Role: FOLLOWER
I20260807 08:47:17.518096 20911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd2e0ea29d14764b8fbf37feb0c6874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40649 } }
I20260807 08:47:17.518194 20911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:17.518234 20911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:17.518278 20911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:17.519217 20911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd2e0ea29d14764b8fbf37feb0c6874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40649 } }
I20260807 08:47:17.519341 20911 leader_election.cc:302] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd2e0ea29d14764b8fbf37feb0c6874; no voters:
I20260807 08:47:17.519558 20911 leader_election.cc:288] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:17.519719 20916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:17.519879 20911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:17.520012 20916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd2e0ea29d14764b8fbf37feb0c6874, State: Running, Role: LEADER
I20260807 08:47:17.520167 20916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd2e0ea29d14764b8fbf37feb0c6874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40649 } }
I20260807 08:47:17.520669 20918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cd2e0ea29d14764b8fbf37feb0c6874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd2e0ea29d14764b8fbf37feb0c6874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40649 } } }
I20260807 08:47:17.520788 20918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:17.520659 20916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cd2e0ea29d14764b8fbf37feb0c6874. Latest consensus state: current_term: 1 leader_uuid: "4cd2e0ea29d14764b8fbf37feb0c6874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd2e0ea29d14764b8fbf37feb0c6874" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 40649 } } }
I20260807 08:47:17.520849 20916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:17.521078 20926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:17.521916 20926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:17.524226 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:17.516231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39911 (local address 127.28.108.190:40649)
0807 08:47:17.516488 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:17.516500 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:17.516548 (+ 48us) server_negotiation.cc:407] Connection header received
0807 08:47:17.517536 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:17.517548 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:17.517623 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:17.517838 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:17.518130 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:17.519643 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:17.521055 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:17.521871 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:17.522049 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:17.523429 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:17.523434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:17.523434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:17.523661 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:17.523703 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:17.523719 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:17.523798 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:17.523892 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 08:47:17.523979 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":128,"threads_started":1}
I20260807 08:47:17.525425 20926 catalog_manager.cc:1347] Generated new cluster ID: faa0b40621794f19a5fbee5f202c9e1e
I20260807 08:47:17.525498 20926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:17.525637 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:17.532562 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:17.527407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55827 (local address 127.28.108.190:40649)
0807 08:47:17.527454 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:17.527457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:17.527618 (+ 161us) server_negotiation.cc:407] Connection header received
0807 08:47:17.527700 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:17.527701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:17.527716 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:17.527769 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:17.529342 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:17.530025 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:17.530948 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:17.531190 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:17.531292 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:17.532109 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:17.532111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:17.532111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:17.532118 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:17.532140 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:17.532146 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:17.532206 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:17.532393 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 08:47:17.532436 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 08:47:17.558589 20926 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:17.559362 20926 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:17.566543 20926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cd2e0ea29d14764b8fbf37feb0c6874: Generated new TSK 0
I20260807 08:47:17.566828 20926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:17.578564 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 20836
2026-08-07T08:47:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1530 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:17 dist-test-slave-9zdj krb5kdc[20944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:17 dist-test-slave-9zdj krb5kdc[20944](info): set up 2 sockets
Aug 07 08:47:17 dist-test-slave-9zdj krb5kdc[20944](info): commencing operation
krb5kdc: starting...
W20260807 08:47:18.721822 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:18 dist-test-slave-9zdj krb5kdc[20944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:47:18.882097 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:32981
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:46325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:18.973194 20958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:18.973477 20958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:18.973526 20958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:18.974853 20958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:18.974937 20958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:18.974963 20958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:18.974984 20958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:18.975015 20958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:18.976698 20958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:46325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:32981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:18.977034 20958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:18.977468 20958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:18.983290 20964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:18.983376 20958 server_base.cc:1034] running on GCE node
W20260807 08:47:18.983276 20967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:18.983278 20969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:18.983807 20958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:18.984158 20958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:18.985360 20958 hybrid_clock.cc:648] HybridClock initialized: now 1786092438985329 us; error 43 us; skew 500 ppm
Aug 07 08:47:18 dist-test-slave-9zdj krb5kdc[20944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092438, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:18.988340 20958 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:47:18.988930 20958 webserver.cc:460] Webserver started at http://127.28.108.190:40201/ using document root <none> and password file <none>
I20260807 08:47:18.989212 20958 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:18.989289 20958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:18.989449 20958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:18.990753 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "d5e73cc7f3084efc9a50e52cb5b07795"
format_stamp: "Formatted at 2026-08-07 08:47:18 on dist-test-slave-9zdj"
server_key: "e70d059475759a01375938721e906bb8"
server_key_iv: "3432087d637811ffde8e3a0e2fbcda59"
server_key_version: "encryptionkey@0"
I20260807 08:47:18.991158 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "d5e73cc7f3084efc9a50e52cb5b07795"
format_stamp: "Formatted at 2026-08-07 08:47:18 on dist-test-slave-9zdj"
server_key: "e70d059475759a01375938721e906bb8"
server_key_iv: "3432087d637811ffde8e3a0e2fbcda59"
server_key_version: "encryptionkey@0"
I20260807 08:47:18.992712 20958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:47:18.993809 20978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:18.994032 20958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:18.994148 20958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "d5e73cc7f3084efc9a50e52cb5b07795"
format_stamp: "Formatted at 2026-08-07 08:47:18 on dist-test-slave-9zdj"
server_key: "e70d059475759a01375938721e906bb8"
server_key_iv: "3432087d637811ffde8e3a0e2fbcda59"
server_key_version: "encryptionkey@0"
I20260807 08:47:18.994243 20958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:18.998010 20958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:19.000972 20958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:19.001199 20958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:19.008359 20958 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:32981
I20260807 08:47:19.008414 21033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:32981 every 8 connection(s)
I20260807 08:47:19.009079 20958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:19.009248 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 20958
I20260807 08:47:19.009408 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:19.009714 29106 external_mini_cluster.cc:1428] Setting key cd272fbe5f5fb02b1d73125834ba4192
I20260807 08:47:19.011510 21034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:19.013135 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:19.010601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59767 (local address 127.28.108.190:32981)
0807 08:47:19.010890 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:19.010910 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:19.010945 (+ 35us) server_negotiation.cc:407] Connection header received
0807 08:47:19.011540 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:19.011549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:19.011581 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:19.011771 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:19.012576 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:19.012581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:19.012581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:19.012586 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:19.012644 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:19.012657 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:19.012797 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:19.012885 (+ 88us) server_negotiation.cc:299] Negotiation successful
0807 08:47:19.012937 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":120,"threads_started":1}
I20260807 08:47:19.014652 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:19.015249 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795: Bootstrap starting.
I20260807 08:47:19.016178 21034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:19.016592 21034 log.cc:824] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:19.017561 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795: No bootstrap required, opened a new log
I20260807 08:47:19.018844 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:19.016809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58173 (local address 127.28.108.190:32981)
0807 08:47:19.016854 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:19.016856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:19.017116 (+ 260us) server_negotiation.cc:407] Connection header received
0807 08:47:19.017718 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:19.017720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:19.017727 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:19.017818 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:19.018590 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:19.018591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:19.018592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 08:47:19.018597 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:19.018615 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 08:47:19.018620 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:19.018701 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:19.018750 (+ 49us) server_negotiation.cc:299] Negotiation successful
0807 08:47:19.018765 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260807 08:47:19.020273 21034 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e73cc7f3084efc9a50e52cb5b07795" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 32981 } }
I20260807 08:47:19.020449 21034 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 08:47:19.020476 21034 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e73cc7f3084efc9a50e52cb5b07795, State: Initialized, Role: FOLLOWER
I20260807 08:47:19.020656 21034 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e73cc7f3084efc9a50e52cb5b07795" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 32981 } }
I20260807 08:47:19.020744 21034 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 08:47:19.020797 21034 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 08:47:19.020855 21034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:19.021693 21034 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e73cc7f3084efc9a50e52cb5b07795" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 32981 } }
I20260807 08:47:19.021836 21034 leader_election.cc:302] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5e73cc7f3084efc9a50e52cb5b07795; no voters:
I20260807 08:47:19.022061 21034 leader_election.cc:288] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 08:47:19.022109 21045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:19.022375 21034 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:19.022418 21045 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [term 1 LEADER]: Becoming Leader. State: Replica: d5e73cc7f3084efc9a50e52cb5b07795, State: Running, Role: LEADER
I20260807 08:47:19.022527 21045 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e73cc7f3084efc9a50e52cb5b07795" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 32981 } }
I20260807 08:47:19.022980 21046 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5e73cc7f3084efc9a50e52cb5b07795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e73cc7f3084efc9a50e52cb5b07795" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 32981 } } }
I20260807 08:47:19.023108 21046 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:19.022979 21047 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5e73cc7f3084efc9a50e52cb5b07795. Latest consensus state: current_term: 1 leader_uuid: "d5e73cc7f3084efc9a50e52cb5b07795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e73cc7f3084efc9a50e52cb5b07795" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 32981 } } }
I20260807 08:47:19.023422 21047 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:19.023411 21052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:19.024044 21052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:19.026280 21052 catalog_manager.cc:1347] Generated new cluster ID: 1f0114b018d44b55a1557660b868c3aa
I20260807 08:47:19.026367 21052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:19.050781 21052 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:19.050993 21052 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:19.055435 21052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5e73cc7f3084efc9a50e52cb5b07795: Generated new TSK 0
I20260807 08:47:19.055689 21052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 08:47:19.063107 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 20958
2026-08-07T08:47:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1481 ms)
[----------] 18 tests from AuthTokenIssuingTest (27141 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:19 dist-test-slave-9zdj krb5kdc[21067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:19 dist-test-slave-9zdj krb5kdc[21067](info): set up 2 sockets
Aug 07 08:47:19 dist-test-slave-9zdj krb5kdc[21067](info): commencing operation
krb5kdc: starting...
W20260807 08:47:20.247546 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:47:20.400302 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:43423
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:43423,127.28.108.189:40033,127.28.108.188:33077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:20.502655 21082 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 08:47:20.502981 21082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:20.503019 21082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:20.503072 21082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:20.504544 21082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:20.504657 21082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:20.504685 21082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:20.504726 21082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:20.504752 21082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:20.506881 21082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:43423,127.28.108.189:40033,127.28.108.188:33077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:43423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:20.507289 21082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:20.507584 21082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:20.513727 21091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:20.513790 21089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:20.513721 21088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:20.514118 21082 server_base.cc:1034] running on GCE node
I20260807 08:47:20.514485 21082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:20.514909 21082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:20.516137 21082 hybrid_clock.cc:648] HybridClock initialized: now 1786092440516051 us; error 106 us; skew 500 ppm
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:20.519579 21082 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:47:20.520152 21082 webserver.cc:460] Webserver started at http://127.28.108.190:38161/ using document root <none> and password file <none>
I20260807 08:47:20.520468 21082 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:20.520524 21082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:20.520677 21082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:20.522066 21082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "f516fe0eba514e2288e6efb3e17ef10b"
format_stamp: "Formatted at 2026-08-07 08:47:20 on dist-test-slave-9zdj"
server_key: "982b3cf6cc0bb2f2d514f2435207eccf"
server_key_iv: "b3ee866c81bdda8ee92bbf5effd744ef"
server_key_version: "encryptionkey@0"
I20260807 08:47:20.522521 21082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "f516fe0eba514e2288e6efb3e17ef10b"
format_stamp: "Formatted at 2026-08-07 08:47:20 on dist-test-slave-9zdj"
server_key: "982b3cf6cc0bb2f2d514f2435207eccf"
server_key_iv: "b3ee866c81bdda8ee92bbf5effd744ef"
server_key_version: "encryptionkey@0"
I20260807 08:47:20.524303 21082 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 08:47:20.525499 21099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:20.525771 21082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:20.526043 21082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "f516fe0eba514e2288e6efb3e17ef10b"
format_stamp: "Formatted at 2026-08-07 08:47:20 on dist-test-slave-9zdj"
server_key: "982b3cf6cc0bb2f2d514f2435207eccf"
server_key_iv: "b3ee866c81bdda8ee92bbf5effd744ef"
server_key_version: "encryptionkey@0"
I20260807 08:47:20.526147 21082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:20.560761 21082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:20.563648 21082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:20.563894 21082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:20.568781 21082 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:43423
I20260807 08:47:20.568851 21157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:43423 every 8 connection(s)
I20260807 08:47:20.569299 21082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:20.570619 21158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:20.571162 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 21082
I20260807 08:47:20.571303 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:20.571517 29106 external_mini_cluster.cc:1428] Setting key b20116dce62198d8ff3ed869782dc6e5
I20260807 08:47:20.573535 21158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.108.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.189@KRBTEST.COM" created.
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:47:20.591212 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.574506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:43423 (local address 127.0.0.1:45108)
0807 08:47:20.574780 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.574837 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:20.575823 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:20.576199 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:20.576203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:20.576670 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:20.576955 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.576964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.578375 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.578377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.579654 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.579658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.580384 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.580385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.580541 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.581176 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:20.581188 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:20.587051 (+ 5863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:20.589307 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.589318 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.589321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.589586 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.589931 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.589932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.589932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.590037 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.590463 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:20.590469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:20.590545 (+ 76us) client_negotiation.cc:769] Sending connection context
0807 08:47:20.590658 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 08:47:20.590937 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":103,"thread_start_us":152,"threads_started":1}
I20260807 08:47:20.591409 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.574608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.28.108.190:43423)
0807 08:47:20.574784 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:20.574794 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:20.575004 (+ 210us) server_negotiation.cc:407] Connection header received
0807 08:47:20.576031 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:20.576034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:20.576054 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:20.576149 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:20.577077 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.578255 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.579793 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.580261 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.580371 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.587186 (+ 6815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:20.587198 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:20.587198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:20.587208 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:20.589162 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.589733 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.589738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.589739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.589849 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.590113 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.590114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.590114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.590385 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:20.590486 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:20.590709 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 08:47:20.590865 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260807 08:47:20.592199 21158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:20.592725 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.592510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40033 (local address 127.0.0.1:38592)
0807 08:47:20.592596 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.592669 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 08:47:20.593000 21101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40033: connect: Connection refused (error 111)
I20260807 08:47:20.593669 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.593499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40033 (local address 127.0.0.1:38602)
0807 08:47:20.593561 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.593603 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260807 08:47:20.593866 21158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.189:40033: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40033: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab.
Entry for principal kudu/127.28.108.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab.
I20260807 08:47:20.611593 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.189:40033
--webserver_interface=127.28.108.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab
--principal=kudu/127.28.108.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:43423,127.28.108.189:40033,127.28.108.188:33077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:47:20.643215 21158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } attempt: 1
I20260807 08:47:20.643978 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.643768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40033 (local address 127.0.0.1:38618)
0807 08:47:20.643880 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.643936 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 08:47:20.644526 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.644345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40033 (local address 127.0.0.1:38628)
0807 08:47:20.644455 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.644496 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260807 08:47:20.644773 21158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.189:40033: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40033: connect: Connection refused (error 111)
W20260807 08:47:20.708098 21165 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 08:47:20.708406 21165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:20.708442 21165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:20.708478 21165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:20.709775 21165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:20.709815 21165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:20.709830 21165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:20.709844 21165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:20.709857 21165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:20.711419 21165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:43423,127.28.108.189:40033,127.28.108.188:33077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab
--principal=kudu/127.28.108.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.189:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.108.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:20.711653 21165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:20.711853 21165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:47:20.713018 21158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } attempt: 2
I20260807 08:47:20.713565 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.713427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40033 (local address 127.0.0.1:38642)
0807 08:47:20.713485 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.713532 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260807 08:47:20.714044 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.713954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40033 (local address 127.0.0.1:38658)
0807 08:47:20.713996 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.714024 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260807 08:47:20.714164 21158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.189:40033: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40033: connect: Connection refused (error 111)
W20260807 08:47:20.719043 21174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:20.718828 21171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:20.719321 21172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:20.719470 21165 server_base.cc:1034] running on GCE node
I20260807 08:47:20.719901 21165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:20.720291 21165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:20.721495 21165 hybrid_clock.cc:648] HybridClock initialized: now 1786092440721470 us; error 43 us; skew 500 ppm
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:20.724875 21165 init.cc:377] Logged in from keytab as kudu/127.28.108.189@KRBTEST.COM (short username kudu)
I20260807 08:47:20.725597 21165 webserver.cc:460] Webserver started at http://127.28.108.189:36487/ using document root <none> and password file <none>
I20260807 08:47:20.725984 21165 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:20.726061 21165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:20.726287 21165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:20.727876 21165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/data/instance:
uuid: "910d44330e6a46df890d4b363150263f"
format_stamp: "Formatted at 2026-08-07 08:47:20 on dist-test-slave-9zdj"
server_key: "4f7695ed2a3504634d1fcd675b3e6611"
server_key_iv: "d51179b735d53b981cb7042e3fc27a0c"
server_key_version: "encryptionkey@0"
I20260807 08:47:20.728360 21165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/wal/instance:
uuid: "910d44330e6a46df890d4b363150263f"
format_stamp: "Formatted at 2026-08-07 08:47:20 on dist-test-slave-9zdj"
server_key: "4f7695ed2a3504634d1fcd675b3e6611"
server_key_iv: "d51179b735d53b981cb7042e3fc27a0c"
server_key_version: "encryptionkey@0"
I20260807 08:47:20.730391 21165 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 08:47:20.731415 21182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:20.731652 21165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 08:47:20.731774 21165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/wal
uuid: "910d44330e6a46df890d4b363150263f"
format_stamp: "Formatted at 2026-08-07 08:47:20 on dist-test-slave-9zdj"
server_key: "4f7695ed2a3504634d1fcd675b3e6611"
server_key_iv: "d51179b735d53b981cb7042e3fc27a0c"
server_key_version: "encryptionkey@0"
I20260807 08:47:20.731865 21165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:20.744380 21165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:20.747430 21165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:20.747658 21165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:20.752893 21165 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.189:40033
I20260807 08:47:20.752928 21241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.189:40033 every 8 connection(s)
I20260807 08:47:20.753407 21165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
I20260807 08:47:20.754688 21242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:20.756633 21242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:20.759460 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 21165
I20260807 08:47:20.759656 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-1/wal/instance
I20260807 08:47:20.759871 29106 external_mini_cluster.cc:1428] Setting key 655cbfc7001f2e496735e74d71144c3b
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:47:20.769371 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.757571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:43423 (local address 127.0.0.1:45114)
0807 08:47:20.757858 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.757901 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:20.758471 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:20.758799 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:20.758803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:20.759087 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:20.759429 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.759434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.760552 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.760554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.761519 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.761521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.762155 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.762157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.762378 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.763098 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:20.763112 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:20.765289 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:20.767600 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.767610 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.767613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.767929 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.768192 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.768193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.768193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.768362 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.768716 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:20.768722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:20.768904 (+ 182us) client_negotiation.cc:769] Sending connection context
0807 08:47:20.768997 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 08:47:20.769157 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":129,"threads_started":1}
I20260807 08:47:20.769357 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.757602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.28.108.190:43423)
0807 08:47:20.757803 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:20.757805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:20.758035 (+ 230us) server_negotiation.cc:407] Connection header received
0807 08:47:20.758610 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:20.758612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:20.758635 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:20.758710 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:20.759560 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.760425 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.761671 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.762032 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.762120 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.765434 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:20.765436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:20.765436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:20.765447 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:20.767476 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.768025 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.768026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.768027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.768111 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.768443 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.768444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.768444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.768635 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:20.768724 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:20.769061 (+ 337us) server_negotiation.cc:299] Negotiation successful
0807 08:47:20.769198 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260807 08:47:20.770403 21242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.108.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.188@KRBTEST.COM" created.
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
I20260807 08:47:20.780683 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.770822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40033 (local address 127.0.0.1:38672)
0807 08:47:20.770891 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.770906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:20.770958 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:20.771339 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:20.771340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:20.771541 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:20.771609 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.771611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.772932 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.772933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.773516 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.773519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.774326 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.774327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.774405 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.775006 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:20.775012 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:20.776681 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:20.779322 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.779324 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.779324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.779525 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.779799 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.779800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.779800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.779865 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.780217 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:20.780218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:20.780263 (+ 45us) client_negotiation.cc:769] Sending connection context
0807 08:47:20.780380 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 08:47:20.780601 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":106}
I20260807 08:47:20.780696 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.770928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.28.108.189:40033)
0807 08:47:20.771176 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:20.771180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:20.771199 (+ 19us) server_negotiation.cc:407] Connection header received
0807 08:47:20.771220 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:20.771222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:20.771234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:20.771323 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:20.771780 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.772825 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.773657 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.774208 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.774327 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.776799 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:20.776816 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:20.776816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:20.776827 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:20.779180 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.779643 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.779658 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.779658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.779723 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.779933 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.779934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.779934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.780140 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:20.780247 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:20.780419 (+ 172us) server_negotiation.cc:299] Negotiation successful
0807 08:47:20.780543 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":155,"threads_started":1}
I20260807 08:47:20.781926 21242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:20.782543 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.782388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54398)
0807 08:47:20.782454 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.782513 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 08:47:20.782702 21187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
I20260807 08:47:20.783438 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.783286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54400)
0807 08:47:20.783367 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.783409 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 08:47:20.783568 21242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:33077: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab.
Entry for principal kudu/127.28.108.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab.
I20260807 08:47:20.798561 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.188:33077
--webserver_interface=127.28.108.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab
--principal=kudu/127.28.108.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:43423,127.28.108.189:40033,127.28.108.188:33077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:47:20.805552 21158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } attempt: 3
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
I20260807 08:47:20.815620 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.806055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40033 (local address 127.0.0.1:38678)
0807 08:47:20.806124 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.806139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:20.806203 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:20.806588 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:20.806589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:20.806762 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:20.806838 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.806840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.808089 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.808090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.808697 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.808700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.809429 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.809430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.809511 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.810226 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:20.810231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:20.811849 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:20.814417 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.814419 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.814420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.814624 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.814937 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.814938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.814938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.815011 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.815333 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:20.815334 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:20.815386 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 08:47:20.815447 (+ 61us) client_negotiation.cc:240] Negotiation successful
0807 08:47:20.815544 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260807 08:47:20.815706 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.806211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.28.108.189:40033)
0807 08:47:20.806292 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:20.806295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:20.806309 (+ 14us) server_negotiation.cc:407] Connection header received
0807 08:47:20.806348 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:20.806350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:20.806366 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:20.806465 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:20.806945 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.807931 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.808853 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.809300 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.809413 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.811977 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:20.811980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:20.811981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:20.811992 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:20.814286 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.814742 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.814743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.814743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.814821 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.815077 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.815078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.815078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.815252 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:20.815330 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:20.815497 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 08:47:20.815596 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 08:47:20.816349 21158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:20.816767 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.816652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54406)
0807 08:47:20.816707 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.816745 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260807 08:47:20.817158 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.817055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54410)
0807 08:47:20.817108 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.817135 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260807 08:47:20.817341 21158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:33077: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
I20260807 08:47:20.832854 21242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } attempt: 1
I20260807 08:47:20.833509 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.833345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54414)
0807 08:47:20.833424 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.833473 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 08:47:20.834089 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.833948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54424)
0807 08:47:20.834022 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.834060 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 08:47:20.834255 21242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:33077: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
I20260807 08:47:20.848572 21158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } attempt: 1
I20260807 08:47:20.849299 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.849092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54430)
0807 08:47:20.849175 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.849243 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 08:47:20.849866 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.849700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54432)
0807 08:47:20.849789 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.849837 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260807 08:47:20.850030 21158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:33077: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
W20260807 08:47:20.901063 21251 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 08:47:20.901366 21251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:20.901399 21251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:20.901471 21251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 08:47:20.902454 21242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } attempt: 2
W20260807 08:47:20.902936 21251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:20.902988 21251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260807 08:47:20.902989 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.902829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54434)
0807 08:47:20.902901 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.902949 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 08:47:20.903013 21251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:20.903038 21251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:20.903079 21251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:20.903477 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.903354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54442)
0807 08:47:20.903413 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.903448 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260807 08:47:20.903587 21242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:33077: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
I20260807 08:47:20.904829 21251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:43423,127.28.108.189:40033,127.28.108.188:33077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab
--principal=kudu/127.28.108.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.188:33077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.108.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:20.905059 21251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:20.905272 21251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:20.911470 21257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:20.911448 21256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:20.911448 21259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:20.911696 21251 server_base.cc:1034] running on GCE node
I20260807 08:47:20.911985 21251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:20.912349 21251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:20.913545 21251 hybrid_clock.cc:648] HybridClock initialized: now 1786092440913524 us; error 44 us; skew 500 ppm
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:20.917035 21251 init.cc:377] Logged in from keytab as kudu/127.28.108.188@KRBTEST.COM (short username kudu)
I20260807 08:47:20.917574 21251 webserver.cc:460] Webserver started at http://127.28.108.188:44439/ using document root <none> and password file <none>
I20260807 08:47:20.917881 21251 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:20.917959 21251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:20.918115 21251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:20.919250 21251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/data/instance:
uuid: "c66af43ccb1c446088e202456a9ad58c"
format_stamp: "Formatted at 2026-08-07 08:47:20 on dist-test-slave-9zdj"
server_key: "6e4d50e76c32a4b3f8935022f9004b86"
server_key_iv: "3535f0142ec42b1a58c9958f6777b721"
server_key_version: "encryptionkey@0"
I20260807 08:47:20.919643 21251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/wal/instance:
uuid: "c66af43ccb1c446088e202456a9ad58c"
format_stamp: "Formatted at 2026-08-07 08:47:20 on dist-test-slave-9zdj"
server_key: "6e4d50e76c32a4b3f8935022f9004b86"
server_key_iv: "3535f0142ec42b1a58c9958f6777b721"
server_key_version: "encryptionkey@0"
I20260807 08:47:20.921140 21251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 08:47:20.922113 21267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:20.922348 21251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 08:47:20.922472 21251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/wal
uuid: "c66af43ccb1c446088e202456a9ad58c"
format_stamp: "Formatted at 2026-08-07 08:47:20 on dist-test-slave-9zdj"
server_key: "6e4d50e76c32a4b3f8935022f9004b86"
server_key_iv: "3535f0142ec42b1a58c9958f6777b721"
server_key_version: "encryptionkey@0"
I20260807 08:47:20.922560 21251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:20.925203 21158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } attempt: 2
I20260807 08:47:20.925767 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.925606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54452)
0807 08:47:20.925689 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.925728 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 08:47:20.926218 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.926079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54456)
0807 08:47:20.926152 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.926186 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 08:47:20.926362 21158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:33077: Network error: Client connection negotiation failed: client connection to 127.28.108.188:33077: connect: Connection refused (error 111)
I20260807 08:47:20.932964 21251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:20.935567 21251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:20.935791 21251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:20.940758 21251 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.188:33077
I20260807 08:47:20.940795 21323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.188:33077 every 8 connection(s)
I20260807 08:47:20.941241 21251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
I20260807 08:47:20.942305 21324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:20.943955 21324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:20.946712 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 21251
I20260807 08:47:20.946821 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786092133523677-29106-0/minicluster-data/master-2/wal/instance
I20260807 08:47:20.947034 29106 external_mini_cluster.cc:1428] Setting key 44677acd46188e99d2b97a08d32a61ac
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
I20260807 08:47:20.954952 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.944886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.28.108.190:43423)
0807 08:47:20.944993 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:20.944995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:20.945239 (+ 244us) server_negotiation.cc:407] Connection header received
0807 08:47:20.945960 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:20.945961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:20.945974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:20.946065 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:20.947016 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.947664 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.948759 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.948968 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.949066 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.951590 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:20.951592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:20.951592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:20.951601 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:20.953170 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.953691 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.953692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.953693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.953759 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.954102 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.954103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.954103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.954288 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:20.954373 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:20.954716 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 08:47:20.954824 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 08:47:20.955273 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.944814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:43423 (local address 127.0.0.1:45124)
0807 08:47:20.945043 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.945095 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:20.945816 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:20.946248 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:20.946253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:20.946554 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:20.946882 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.946889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.947765 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.947766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.948652 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.948655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.949076 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.949077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.949236 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.949872 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:20.949881 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:20.951493 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:20.953286 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.953295 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.953297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.953558 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.953835 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.953836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.953837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.954003 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.954376 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:20.954390 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:20.954531 (+ 141us) client_negotiation.cc:769] Sending connection context
0807 08:47:20.954668 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 08:47:20.954868 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":132,"threads_started":1}
I20260807 08:47:20.956051 21324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:20.957765 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.947528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.28.108.188:33077)
0807 08:47:20.947713 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:20.947719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:20.947847 (+ 128us) server_negotiation.cc:407] Connection header received
0807 08:47:20.947913 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:20.947917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:20.947941 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:20.948032 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:20.948720 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.949449 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.950336 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.950783 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.950866 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.953666 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:20.953676 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:20.953677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:20.953684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:20.955980 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.956390 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.956393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.956393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.956463 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.956869 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.956870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.956871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.957097 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:20.957262 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:20.957485 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 08:47:20.957563 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:47:20.964442 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.956624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.28.108.189:40033)
0807 08:47:20.956700 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:20.956702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:20.956712 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:47:20.956741 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:20.956742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:20.956753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:20.956811 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:20.957194 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.958005 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.958942 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.959186 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.959266 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.961408 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:20.961410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:20.961410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:20.961418 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:20.963197 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.963597 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.963599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.963600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.963665 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.963890 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.963891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.963891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.964014 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:20.964092 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:20.964262 (+ 170us) server_negotiation.cc:299] Negotiation successful
0807 08:47:20.964338 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 08:47:20.964442 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.956542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40033 (local address 127.0.0.1:38684)
0807 08:47:20.956595 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.956606 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:20.956667 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:20.956812 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:20.956813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:20.956993 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:20.957067 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.957069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.958109 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.958110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.958857 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.958859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.959273 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.959274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.959353 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.959863 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:20.959868 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:20.961299 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:20.963299 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.963300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.963301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.963526 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.963740 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.963741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.963741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.963815 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.964084 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:20.964085 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:20.964135 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 08:47:20.964227 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 08:47:20.964338 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260807 08:47:20.965234 21324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:20.966966 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.958664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.28.108.190:43423)
0807 08:47:20.958708 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:20.958710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:20.958719 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:47:20.958742 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:20.958743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:20.958756 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:20.958812 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:20.959405 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.960092 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.960868 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.961255 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.961321 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.963903 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:20.963904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:20.963904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:20.963912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:20.965678 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.966085 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.966086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.966087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.966161 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.966383 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.966384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.966384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.966517 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:20.966741 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:20.966780 (+ 39us) server_negotiation.cc:299] Negotiation successful
0807 08:47:20.966861 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
I20260807 08:47:20.973551 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.965662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54470)
0807 08:47:20.965728 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.965741 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:20.965796 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:20.965922 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:20.965922 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:20.966065 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:20.966104 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.966105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.966953 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.966954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.967990 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.967993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.968577 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.968579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.968641 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.969079 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:20.969085 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:20.970773 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:20.972439 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.972440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.972440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.972621 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.972827 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:20.972828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:20.972828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:20.972886 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:20.973186 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:20.973187 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:20.973229 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 08:47:20.973318 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 08:47:20.973427 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260807 08:47:20.973564 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.965768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.28.108.188:33077)
0807 08:47:20.965816 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:20.965818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:20.965826 (+ 8us) server_negotiation.cc:407] Connection header received
0807 08:47:20.965849 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:20.965849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:20.965862 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:20.965918 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:20.966197 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.966773 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.968074 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.968455 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.968541 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.970884 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:20.970885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:20.970885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:20.970892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:20.972324 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.972697 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.972698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.972698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.972758 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.972958 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.972959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.972959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.973106 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:20.973197 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:20.973344 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 08:47:20.973474 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":24}
I20260807 08:47:20.975370 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c: Bootstrap starting.
I20260807 08:47:20.976266 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.967783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.28.108.189:40033)
0807 08:47:20.967838 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:20.967840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:20.967849 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:47:20.967881 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:20.967882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:20.967892 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:20.967957 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:20.968575 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.969418 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.970707 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.970921 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.971012 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.973434 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:20.973436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:20.973436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:20.973444 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:20.975026 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.975420 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.975421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.975421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.975490 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:20.975690 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:20.975692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:20.975692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:20.975821 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:20.976038 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:20.976077 (+ 39us) server_negotiation.cc:299] Negotiation successful
0807 08:47:20.976162 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260807 08:47:20.976364 21324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:20.976747 21324 log.cc:824] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:20.977979 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c: No bootstrap required, opened a new log
I20260807 08:47:20.978039 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:20.979943 21324 raft_consensus.cc:348] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } }
I20260807 08:47:20.980088 21324 raft_consensus.cc:729] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66af43ccb1c446088e202456a9ad58c, State: Initialized, Role: FOLLOWER
I20260807 08:47:20.980185 21324 consensus_queue.cc:260] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } }
I20260807 08:47:20.979568 21130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f516fe0eba514e2288e6efb3e17ef10b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45128
I20260807 08:47:20.980561 21324 sys_catalog.cc:564] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:20.980513 21330 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } } }
I20260807 08:47:20.980748 21330 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [sys.catalog]: This master's current role is: FOLLOWER
W20260807 08:47:20.982787 21341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:47:20.982859 21341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:47:20.983098 21251 master_runner.cc:186] Error getting master registration for 127.28.108.190:43423: OK, instance_id { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" instance_seqno: 1786092440567340 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 08:47:20.986368 21251 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 08:47:20.994745 21242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } attempt: 3
Aug 07 08:47:20 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
I20260807 08:47:21.003319 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.995215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54476)
0807 08:47:20.995318 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:20.995334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:20.995410 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:20.995577 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:20.995578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:20.995774 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:20.995825 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.995826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.996510 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.996512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.997038 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:20.997040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.997453 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.997455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:20.997516 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:20.998081 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:20.998088 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:20.999912 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:21.001950 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:21.001951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:21.001952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:21.002226 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:21.002477 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:21.002478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:21.002478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:21.002555 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:21.002933 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:21.002934 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:21.002995 (+ 61us) client_negotiation.cc:769] Sending connection context
0807 08:47:21.003073 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 08:47:21.003207 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 08:47:21.003317 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:20.995301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.28.108.188:33077)
0807 08:47:20.995349 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:20.995351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:20.995389 (+ 38us) server_negotiation.cc:407] Connection header received
0807 08:47:20.995477 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:20.995478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:20.995491 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:20.995570 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:20.995930 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.996414 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.997159 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:20.997361 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:20.997444 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:21.000040 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:21.000041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:21.000041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:21.000051 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:21.001837 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:21.002316 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:21.002318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:21.002318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:21.002393 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:21.002621 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:21.002622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:21.002622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:21.002855 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:21.002925 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:21.003109 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 08:47:21.003206 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 08:47:21.005793 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f: Bootstrap starting.
I20260807 08:47:21.006522 21242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:21.006906 21242 log.cc:824] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:21.007773 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f: No bootstrap required, opened a new log
I20260807 08:47:21.009310 21242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } }
I20260807 08:47:21.009474 21242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910d44330e6a46df890d4b363150263f, State: Initialized, Role: FOLLOWER
I20260807 08:47:21.009582 21242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } }
I20260807 08:47:21.009949 21344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } } }
I20260807 08:47:21.010031 21242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:21.010042 21344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [sys.catalog]: This master's current role is: FOLLOWER
W20260807 08:47:21.012017 21356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:47:21.012099 21356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:47:21.012367 21165 master_runner.cc:186] Error getting master registration for 127.28.108.190:43423: OK, instance_id { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" instance_seqno: 1786092440567340 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 08:47:21.015491 21165 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 08:47:21.025456 21158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } attempt: 3
Aug 07 08:47:21 dist-test-slave-9zdj krb5kdc[21067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092440, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
I20260807 08:47:21.034973 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:21.026006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.28.108.188:33077)
0807 08:47:21.026084 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.026086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.026094 (+ 8us) server_negotiation.cc:407] Connection header received
0807 08:47:21.026167 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.026168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.026178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.026236 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.026645 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:21.027396 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:21.028179 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:21.028480 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:21.028572 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:21.031459 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:21.031460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:21.031460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:21.031471 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:21.033524 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:21.033987 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:21.033988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:21.033989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:21.034059 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:21.034284 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:21.034285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:21.034285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:21.034495 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:21.034559 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:21.034780 (+ 221us) server_negotiation.cc:299] Negotiation successful
0807 08:47:21.034871 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 08:47:21.035020 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:21.025860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:33077 (local address 127.0.0.1:54492)
0807 08:47:21.025923 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:21.025936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:21.025984 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:21.026269 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:21.026270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:21.026485 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:21.026548 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:21.026551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:21.027474 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:21.027475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:21.028064 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:21.028067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:21.028584 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:21.028585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:21.028644 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:21.029376 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:21.029382 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:21.031308 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:21.033675 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:21.033677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:21.033677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:21.033892 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:21.034139 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:21.034140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:21.034140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:21.034209 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:21.034573 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:21.034574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:21.034629 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 08:47:21.034717 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 08:47:21.034910 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260807 08:47:21.037252 21158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b: Bootstrap starting.
I20260807 08:47:21.038167 21158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:21.038506 21158 log.cc:824] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:21.039412 21158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b: No bootstrap required, opened a new log
I20260807 08:47:21.040894 21158 raft_consensus.cc:348] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } }
I20260807 08:47:21.041064 21158 raft_consensus.cc:729] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f516fe0eba514e2288e6efb3e17ef10b, State: Initialized, Role: FOLLOWER
I20260807 08:47:21.041168 21158 consensus_queue.cc:260] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } }
I20260807 08:47:21.041539 21158 sys_catalog.cc:564] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:21.041664 21130 raft_consensus.cc:482] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:47:21.041719 21130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:21.042685 21130 raft_consensus.cc:504] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } }
I20260807 08:47:21.043421 21212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f516fe0eba514e2288e6efb3e17ef10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "910d44330e6a46df890d4b363150263f"
I20260807 08:47:21.043489 21294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f516fe0eba514e2288e6efb3e17ef10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c66af43ccb1c446088e202456a9ad58c"
I20260807 08:47:21.043583 21212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:21.043606 21294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:21.043002 21130 leader_election.cc:288] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [CANDIDATE]: Term 1 election: Requested vote from peers 910d44330e6a46df890d4b363150263f (127.28.108.189:40033), c66af43ccb1c446088e202456a9ad58c (127.28.108.188:33077)
I20260807 08:47:21.044572 21294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f516fe0eba514e2288e6efb3e17ef10b in term 1.
I20260807 08:47:21.044574 21212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f516fe0eba514e2288e6efb3e17ef10b in term 1.
I20260807 08:47:21.044855 21101 leader_election.cc:302] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910d44330e6a46df890d4b363150263f, f516fe0eba514e2288e6efb3e17ef10b; no voters:
I20260807 08:47:21.041498 21360 sys_catalog.cc:455] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } } }
I20260807 08:47:21.045243 21360 sys_catalog.cc:458] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:47:21.045361 21360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:21.045454 21360 raft_consensus.cc:686] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [term 1 LEADER]: Becoming Leader. State: Replica: f516fe0eba514e2288e6efb3e17ef10b, State: Running, Role: LEADER
I20260807 08:47:21.045588 21360 consensus_queue.cc:237] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } }
W20260807 08:47:21.045842 21371 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:47:21.045893 21371 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:47:21.047806 21082 master_runner.cc:263] No leader master found from master f516fe0eba514e2288e6efb3e17ef10b
I20260807 08:47:21.047858 21082 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 08:47:21.049897 21373 sys_catalog.cc:455] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f516fe0eba514e2288e6efb3e17ef10b. Latest consensus state: current_term: 1 leader_uuid: "f516fe0eba514e2288e6efb3e17ef10b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } } }
I20260807 08:47:21.050025 21373 sys_catalog.cc:458] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:21.050335 21379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:21.050520 21379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:21.058934 21294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [term 1 FOLLOWER]: Refusing update from remote peer f516fe0eba514e2288e6efb3e17ef10b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:47:21.059470 21373 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:47:21.061167 21330 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f516fe0eba514e2288e6efb3e17ef10b. Latest consensus state: current_term: 1 leader_uuid: "f516fe0eba514e2288e6efb3e17ef10b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } } }
I20260807 08:47:21.061285 21330 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:47:21.061945 21212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [term 1 FOLLOWER]: Refusing update from remote peer f516fe0eba514e2288e6efb3e17ef10b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:47:21.062331 21373 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [LEADER]: Connected to new peer: Peer: permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:47:21.064299 21344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f516fe0eba514e2288e6efb3e17ef10b. Latest consensus state: current_term: 1 leader_uuid: "f516fe0eba514e2288e6efb3e17ef10b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } } }
I20260807 08:47:21.064409 21344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [sys.catalog]: This master's current role is: FOLLOWER
W20260807 08:47:21.064973 21105 tablet.cc:2369] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 08:47:21.066103 21373 sys_catalog.cc:455] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f516fe0eba514e2288e6efb3e17ef10b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } } }
I20260807 08:47:21.066205 21373 sys_catalog.cc:458] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:21.066709 21344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f516fe0eba514e2288e6efb3e17ef10b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } } }
I20260807 08:47:21.067143 21344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:47:21.067503 21381 mvcc.cc:205] Tried to move back new op lower bound from 7315834638573817856 to 7315834638524477440. Current Snapshot: MvccSnapshot[applied={T|T < 7315834638573817856}]
I20260807 08:47:21.068135 21389 mvcc.cc:205] Tried to move back new op lower bound from 7315834638573817856 to 7315834638524477440. Current Snapshot: MvccSnapshot[applied={T|T < 7315834638573817856}]
I20260807 08:47:21.070892 21360 sys_catalog.cc:455] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f516fe0eba514e2288e6efb3e17ef10b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } } }
I20260807 08:47:21.071017 21360 sys_catalog.cc:458] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:21.071518 21330 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f516fe0eba514e2288e6efb3e17ef10b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516fe0eba514e2288e6efb3e17ef10b" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 43423 } } peers { permanent_uuid: "910d44330e6a46df890d4b363150263f" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40033 } } peers { permanent_uuid: "c66af43ccb1c446088e202456a9ad58c" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 33077 } } }
I20260807 08:47:21.071621 21330 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:47:21.071722 21385 mvcc.cc:205] Tried to move back new op lower bound from 7315834638573817856 to 7315834638524477440. Current Snapshot: MvccSnapshot[applied={T|T < 7315834638573817856}]
I20260807 08:47:21.072355 21379 catalog_manager.cc:1347] Generated new cluster ID: 4014bcfb84db4b86a3ead912b5ce841b
I20260807 08:47:21.072414 21379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:21.072809 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:21.053944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.28.108.188:33077)
0807 08:47:21.053996 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.053998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.061835 (+ 7837us) server_negotiation.cc:407] Connection header received
0807 08:47:21.061950 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.061951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.061963 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.061996 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.062960 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:21.063743 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:21.066067 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:21.066394 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:21.066437 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:21.068749 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:21.068751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:21.068751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:21.068759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:21.070585 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:21.071330 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:21.071331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:21.071332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:21.071399 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:21.072012 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:21.072013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:21.072014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:21.072153 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:21.072228 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:21.072622 (+ 394us) server_negotiation.cc:299] Negotiation successful
0807 08:47:21.072718 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 08:47:21.079161 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:21.054778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.28.108.190:43423)
0807 08:47:21.054838 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.054840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.069834 (+ 14994us) server_negotiation.cc:407] Connection header received
0807 08:47:21.069946 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.069947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.069959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.069989 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.072557 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:21.073306 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:21.074263 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:21.074562 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:21.074627 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:21.076077 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:21.076078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:21.076079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:21.076086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:21.077793 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:21.078281 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:21.078283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:21.078283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:21.078346 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:21.078527 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:21.078527 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:21.078528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:21.078674 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:21.078763 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:21.078967 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 08:47:21.079058 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 08:47:21.081264 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:21.054403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.28.108.189:40033)
0807 08:47:21.054449 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.054450 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.064971 (+ 10521us) server_negotiation.cc:407] Connection header received
0807 08:47:21.065073 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.065074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.065086 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.065149 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.066836 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:21.067571 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:21.070666 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:21.070951 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:21.071018 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:21.077765 (+ 6747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:21.077768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:21.077768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:21.077780 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:21.079715 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:21.080238 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:21.080241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:21.080241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:21.080315 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:21.080599 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:21.080601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:21.080601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:21.080765 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:21.080858 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:21.081057 (+ 199us) server_negotiation.cc:299] Negotiation successful
0807 08:47:21.081152 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 08:47:21.098383 21379 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:21.099255 21379 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:21.105522 21379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f516fe0eba514e2288e6efb3e17ef10b: Generated new TSK 0
I20260807 08:47:21.105867 21379 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 08:47:21.136734 21328 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.134938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.28.108.188:33077)
0807 08:47:21.134992 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.134995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.135833 (+ 838us) server_negotiation.cc:407] Connection header received
0807 08:47:21.135963 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.135965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.135980 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.136017 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.136644 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54504: BlockingRecv error: recv got EOF from 127.0.0.1:54504 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260807 08:47:21.137692 21247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.135480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.28.108.189:40033)
0807 08:47:21.135533 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.135535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.137034 (+ 1499us) server_negotiation.cc:407] Connection header received
0807 08:47:21.137159 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.137161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.137174 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.137209 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.137597 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38712: BlockingRecv error: recv got EOF from 127.0.0.1:38712 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260807 08:47:21.138902 21162 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.138112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.28.108.190:43423)
0807 08:47:21.138177 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.138179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.138322 (+ 143us) server_negotiation.cc:407] Connection header received
0807 08:47:21.138430 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.138431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.138443 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.138476 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.138830 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45152: BlockingRecv error: recv got EOF from 127.0.0.1:45152 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260807 08:47:21.141855 21247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.140833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.28.108.189:40033)
0807 08:47:21.140880 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.140882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.141135 (+ 253us) server_negotiation.cc:407] Connection header received
0807 08:47:21.141272 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.141273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.141295 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.141331 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.141788 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38722: BlockingRecv error: recv got EOF from 127.0.0.1:38722 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260807 08:47:21.145534 21247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.144490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.28.108.189:40033)
0807 08:47:21.144532 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.144534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.144852 (+ 318us) server_negotiation.cc:407] Connection header received
0807 08:47:21.144970 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.144971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.144984 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.145017 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.145468 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38728: BlockingRecv error: recv got EOF from 127.0.0.1:38728 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260807 08:47:21.150483 21247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.149355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.28.108.189:40033)
0807 08:47:21.149407 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.149410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.149758 (+ 348us) server_negotiation.cc:407] Connection header received
0807 08:47:21.149879 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.149880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.149893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.149926 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.150408 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38732: BlockingRecv error: recv got EOF from 127.0.0.1:38732 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260807 08:47:21.157218 21247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.156118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.28.108.189:40033)
0807 08:47:21.156186 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.156188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.156511 (+ 323us) server_negotiation.cc:407] Connection header received
0807 08:47:21.156642 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.156643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.156657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.156691 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.157150 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38742: BlockingRecv error: recv got EOF from 127.0.0.1:38742 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260807 08:47:21.168056 21247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.166891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.28.108.189:40033)
0807 08:47:21.166947 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.166950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.167300 (+ 350us) server_negotiation.cc:407] Connection header received
0807 08:47:21.167425 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.167426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.167440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.167474 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.167971 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38746: BlockingRecv error: recv got EOF from 127.0.0.1:38746 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260807 08:47:21.187310 21247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.185868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.28.108.189:40033)
0807 08:47:21.185922 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.185925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.186288 (+ 363us) server_negotiation.cc:407] Connection header received
0807 08:47:21.186413 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.186414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.186427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.186480 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.187241 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38754: BlockingRecv error: recv got EOF from 127.0.0.1:38754 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260807 08:47:21.222147 21247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.221121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.28.108.189:40033)
0807 08:47:21.221240 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.221243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.221420 (+ 177us) server_negotiation.cc:407] Connection header received
0807 08:47:21.221512 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.221514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.221535 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.221611 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.222050 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38770: BlockingRecv error: recv got EOF from 127.0.0.1:38770 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260807 08:47:21.288518 21247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.287563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.28.108.189:40033)
0807 08:47:21.287671 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.287673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.287770 (+ 97us) server_negotiation.cc:407] Connection header received
0807 08:47:21.287884 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.287886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.287901 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.287989 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.288420 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38774: BlockingRecv error: recv got EOF from 127.0.0.1:38774 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 08:47:21.418900 21247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.418073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38780 (local address 127.28.108.189:40033)
0807 08:47:21.418184 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.418186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.418301 (+ 115us) server_negotiation.cc:407] Connection header received
0807 08:47:21.418340 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.418342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.418355 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.418442 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.418845 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38780: BlockingRecv error: recv got EOF from 127.0.0.1:38780 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260807 08:47:21.677846 21247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:21.676871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.28.108.189:40033)
0807 08:47:21.676983 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:21.676985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:21.677027 (+ 42us) server_negotiation.cc:407] Connection header received
0807 08:47:21.677069 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:21.677070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:21.677091 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:21.677141 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:21.677752 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38786: BlockingRecv error: recv got EOF from 127.0.0.1:38786 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 08:47:21.983361 21341 catalog_manager.cc:1259] Loaded cluster ID: 4014bcfb84db4b86a3ead912b5ce841b
I20260807 08:47:21.983459 21341 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c: loading cluster ID for follower catalog manager: success
I20260807 08:47:21.984527 21341 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c: acquiring CA information for follower catalog manager: success
I20260807 08:47:21.984860 21341 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c66af43ccb1c446088e202456a9ad58c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 08:47:22.012753 21356 catalog_manager.cc:1259] Loaded cluster ID: 4014bcfb84db4b86a3ead912b5ce841b
I20260807 08:47:22.012863 21356 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f: loading cluster ID for follower catalog manager: success
I20260807 08:47:22.013883 21356 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f: acquiring CA information for follower catalog manager: success
I20260807 08:47:22.014281 21356 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 910d44330e6a46df890d4b363150263f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 08:47:22.194454 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:22.191232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.28.108.189:40033)
0807 08:47:22.191364 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:22.191366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:22.191490 (+ 124us) server_negotiation.cc:407] Connection header received
0807 08:47:22.191547 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:22.191548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:22.191621 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:22.191696 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:47:22.191866 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:22.192528 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:22.193429 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:22.193673 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:22.193743 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:22.193929 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:47:22.194057 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:47:22.194123 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:22.194165 (+ 42us) server_negotiation.cc:299] Negotiation successful
0807 08:47:22.194188 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260807 08:47:22.200502 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:22.197154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.28.108.188:33077)
0807 08:47:22.197329 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:22.197332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:22.197343 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:47:22.197449 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:22.197451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:22.197509 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:22.197659 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:47:22.197901 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:22.198492 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:22.199358 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:22.199605 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:22.199698 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:22.199912 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:47:22.200051 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:47:22.200157 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:22.200263 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 08:47:22.200292 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260807 08:47:22.205768 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:22.202761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.28.108.188:33077)
0807 08:47:22.202803 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:22.202805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:22.202817 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:47:22.202852 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:22.202852 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:22.202867 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:22.202933 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:47:22.203094 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:22.203838 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:22.204862 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:22.205076 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:22.205210 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:22.205431 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:47:22.205471 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:47:22.205501 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:22.205643 (+ 142us) server_negotiation.cc:299] Negotiation successful
0807 08:47:22.205673 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 08:47:22.206612 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:22.202640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.28.108.189:40033)
0807 08:47:22.202733 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:22.202736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:22.202928 (+ 192us) server_negotiation.cc:407] Connection header received
0807 08:47:22.203045 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:22.203047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:22.203062 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:22.203112 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:47:22.203426 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:22.204109 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:22.205230 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:22.205490 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:22.205525 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:22.205766 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:47:22.205807 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:47:22.205883 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:22.206484 (+ 601us) server_negotiation.cc:299] Negotiation successful
0807 08:47:22.206522 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 08:47:22.206909 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:22.202789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.28.108.190:43423)
0807 08:47:22.202908 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:22.202910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:22.202919 (+ 9us) server_negotiation.cc:407] Connection header received
0807 08:47:22.203042 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:22.203043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:22.203106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:22.203219 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:47:22.203545 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:22.204268 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:22.205357 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:22.205749 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:22.205823 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:22.206158 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:47:22.206302 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:47:22.206407 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:22.206741 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 08:47:22.206780 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
I20260807 08:47:22.209658 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 21082
I20260807 08:47:22.219403 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 21165
I20260807 08:47:22.228394 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 21251
2026-08-07T08:47:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3168 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 08:47:22 dist-test-slave-9zdj krb5kdc[21485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 08:47:22 dist-test-slave-9zdj krb5kdc[21485](info): set up 2 sockets
Aug 07 08:47:22 dist-test-slave-9zdj krb5kdc[21485](info): commencing operation
krb5kdc: starting...
W20260807 08:47:23.396237 29106 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/test-user.keytab.
Aug 07 08:47:23 dist-test-slave-9zdj krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T08:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T08:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.108.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
Entry for principal kudu/127.28.108.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab.
I20260807 08:47:23.554363 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:41913
--webserver_interface=127.28.108.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:42133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:41913,127.28.108.189:40751,127.28.108.188:38965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 08:47:23.643482 21501 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 08:47:23.643767 21501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:23.643788 21501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:23.643813 21501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:23.645192 21501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:23.645251 21501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:23.645267 21501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:23.645279 21501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:23.645292 21501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:23.647239 21501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:42133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:41913,127.28.108.189:40751,127.28.108.188:38965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.190.keytab
--principal=kudu/127.28.108.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.190:41913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.108.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:23.647536 21501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:23.647760 21501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:23.654042 21506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:23.654042 21509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:23.654071 21507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:23.654893 21501 server_base.cc:1034] running on GCE node
I20260807 08:47:23.655268 21501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:23.655680 21501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:23.656924 21501 hybrid_clock.cc:648] HybridClock initialized: now 1786092443656921 us; error 78 us; skew 500 ppm
Aug 07 08:47:23 dist-test-slave-9zdj krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:23.660494 21501 init.cc:377] Logged in from keytab as kudu/127.28.108.190@KRBTEST.COM (short username kudu)
I20260807 08:47:23.661167 21501 webserver.cc:460] Webserver started at http://127.28.108.190:45791/ using document root <none> and password file <none>
I20260807 08:47:23.661484 21501 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:23.661543 21501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:23.661751 21501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:23.663069 21501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/data/instance:
uuid: "f33dbf21d611473dac4245d89acf9558"
format_stamp: "Formatted at 2026-08-07 08:47:23 on dist-test-slave-9zdj"
server_key: "999c2f2e9644957d745d98b0e90a387d"
server_key_iv: "cbeb579300261d41ad9f66525ee6b36c"
server_key_version: "encryptionkey@0"
I20260807 08:47:23.663491 21501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/wal/instance:
uuid: "f33dbf21d611473dac4245d89acf9558"
format_stamp: "Formatted at 2026-08-07 08:47:23 on dist-test-slave-9zdj"
server_key: "999c2f2e9644957d745d98b0e90a387d"
server_key_iv: "cbeb579300261d41ad9f66525ee6b36c"
server_key_version: "encryptionkey@0"
I20260807 08:47:23.665362 21501 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260807 08:47:23.666604 21516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:23.666916 21501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:23.667050 21501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/wal
uuid: "f33dbf21d611473dac4245d89acf9558"
format_stamp: "Formatted at 2026-08-07 08:47:23 on dist-test-slave-9zdj"
server_key: "999c2f2e9644957d745d98b0e90a387d"
server_key_iv: "cbeb579300261d41ad9f66525ee6b36c"
server_key_version: "encryptionkey@0"
I20260807 08:47:23.667147 21501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:23.677795 21501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:23.680997 21501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:23.681284 21501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:23.686931 21501 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.190:41913
I20260807 08:47:23.687024 21574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.190:41913 every 8 connection(s)
I20260807 08:47:23.687496 21501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/data/info.pb
I20260807 08:47:23.688758 21575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:23.690768 21575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:23.692080 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 21501
I20260807 08:47:23.692212 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-0/wal/instance
I20260807 08:47:23.692477 29106 external_mini_cluster.cc:1428] Setting key b3b60504bc6ebf575e77b29ac3201257
Aug 07 08:47:23 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.108.189@KRBTEST.COM; defaulting to no policy
I20260807 08:47:23.703706 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.691673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.28.108.190:41913)
0807 08:47:23.691923 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:23.691935 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:23.692118 (+ 183us) server_negotiation.cc:407] Connection header received
0807 08:47:23.693041 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:23.693044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:23.693074 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:23.693193 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:23.693968 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.694913 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.695998 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.696391 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.696641 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:23.699538 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:23.699549 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:23.699549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:23.699559 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:23.701752 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:23.702255 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:23.702259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:23.702259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:23.702377 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:23.702706 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:23.702707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:23.702708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:23.702998 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:23.703122 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:23.703378 (+ 256us) server_negotiation.cc:299] Negotiation successful
0807 08:47:23.703503 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":70,"threads_started":1}
I20260807 08:47:23.703873 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.691575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41913 (local address 127.0.0.1:51296)
0807 08:47:23.691915 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.691980 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:23.692842 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:23.693203 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:23.693206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:23.693558 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:23.693846 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:23.693854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.695056 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.695057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:23.695883 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:23.695886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.696589 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.696590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:23.696669 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:23.697368 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:23.697377 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:23.699404 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:23.701870 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:23.701878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:23.701880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:23.702163 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:23.702470 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:23.702471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:23.702471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:23.702579 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:23.703094 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:23.703110 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:23.703214 (+ 104us) client_negotiation.cc:769] Sending connection context
0807 08:47:23.703336 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 08:47:23.703661 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":130,"thread_start_us":150,"threads_started":1}
I20260807 08:47:23.704840 21575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:23.705730 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.705425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40751 (local address 127.0.0.1:57084)
0807 08:47:23.705559 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.705691 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260807 08:47:23.705902 21517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40751: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.189@KRBTEST.COM" created.
I20260807 08:47:23.706612 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.706473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40751 (local address 127.0.0.1:57092)
0807 08:47:23.706533 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.706573 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 08:47:23.706782 21575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.189:40751: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40751: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab.
Entry for principal kudu/127.28.108.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab.
I20260807 08:47:23.726742 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.189:40751
--webserver_interface=127.28.108.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab
--principal=kudu/127.28.108.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:42133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:41913,127.28.108.189:40751,127.28.108.188:38965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:47:23.756106 21575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } attempt: 1
I20260807 08:47:23.756799 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.756622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40751 (local address 127.0.0.1:57106)
0807 08:47:23.756705 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.756758 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 08:47:23.757304 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.757193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40751 (local address 127.0.0.1:57120)
0807 08:47:23.757241 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.757281 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260807 08:47:23.757570 21575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.189:40751: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40751: connect: Connection refused (error 111)
W20260807 08:47:23.818293 21582 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 08:47:23.818614 21582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:23.818648 21582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:23.818697 21582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:23.820044 21582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:23.820144 21582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:23.820166 21582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:23.820192 21582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:23.820225 21582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:23.822021 21582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:42133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:41913,127.28.108.189:40751,127.28.108.188:38965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.189.keytab
--principal=kudu/127.28.108.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.189:40751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.108.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:23.822371 21582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:23.822645 21582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 08:47:23.825789 21575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } attempt: 2
I20260807 08:47:23.826591 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.826164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40751 (local address 127.0.0.1:57126)
0807 08:47:23.826460 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.826509 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260807 08:47:23.827494 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.827038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40751 (local address 127.0.0.1:57140)
0807 08:47:23.827413 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.827452 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
W20260807 08:47:23.827692 21575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.189:40751: Network error: Client connection negotiation failed: client connection to 127.28.108.189:40751: connect: Connection refused (error 111)
I20260807 08:47:23.829262 21582 server_base.cc:1034] running on GCE node
W20260807 08:47:23.829209 21589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:23.829207 21587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:23.829209 21591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:23.829813 21582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:23.830195 21582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:23.831391 21582 hybrid_clock.cc:648] HybridClock initialized: now 1786092443831364 us; error 42 us; skew 500 ppm
Aug 07 08:47:23 dist-test-slave-9zdj krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:23.835230 21582 init.cc:377] Logged in from keytab as kudu/127.28.108.189@KRBTEST.COM (short username kudu)
I20260807 08:47:23.835973 21582 webserver.cc:460] Webserver started at http://127.28.108.189:42091/ using document root <none> and password file <none>
I20260807 08:47:23.836410 21582 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:23.836498 21582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:23.836709 21582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:23.838377 21582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/data/instance:
uuid: "b8455ceb81d3428e988d8336bedcaa60"
format_stamp: "Formatted at 2026-08-07 08:47:23 on dist-test-slave-9zdj"
server_key: "5464d30bca4cbfd88ba31379e61f1c24"
server_key_iv: "b0285b14a0d2e058b6e7d89172262bd4"
server_key_version: "encryptionkey@0"
I20260807 08:47:23.838804 21582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/wal/instance:
uuid: "b8455ceb81d3428e988d8336bedcaa60"
format_stamp: "Formatted at 2026-08-07 08:47:23 on dist-test-slave-9zdj"
server_key: "5464d30bca4cbfd88ba31379e61f1c24"
server_key_iv: "b0285b14a0d2e058b6e7d89172262bd4"
server_key_version: "encryptionkey@0"
I20260807 08:47:23.840440 21582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 08:47:23.841676 21598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:23.841938 21582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:23.842067 21582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/wal
uuid: "b8455ceb81d3428e988d8336bedcaa60"
format_stamp: "Formatted at 2026-08-07 08:47:23 on dist-test-slave-9zdj"
server_key: "5464d30bca4cbfd88ba31379e61f1c24"
server_key_iv: "b0285b14a0d2e058b6e7d89172262bd4"
server_key_version: "encryptionkey@0"
I20260807 08:47:23.842165 21582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:23.861052 21582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:23.863943 21582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:23.864130 21582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:23.868685 21582 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.189:40751
I20260807 08:47:23.868731 21654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.189:40751 every 8 connection(s)
I20260807 08:47:23.869207 21582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/data/info.pb
I20260807 08:47:23.870535 21655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:23.872460 21655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:23.874174 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 21582
I20260807 08:47:23.874307 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-1/wal/instance
I20260807 08:47:23.874562 29106 external_mini_cluster.cc:1428] Setting key 7e4ef921e06695f2a1893953cc35360e
Aug 07 08:47:23 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.108.188@KRBTEST.COM; defaulting to no policy
I20260807 08:47:23.885509 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.873681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.28.108.190:41913)
0807 08:47:23.873755 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:23.873757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:23.873951 (+ 194us) server_negotiation.cc:407] Connection header received
0807 08:47:23.874653 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:23.874655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:23.874674 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:23.874770 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:23.875748 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.876613 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.878606 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.878991 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.879084 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:23.881896 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:23.881901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:23.881901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:23.881915 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:23.883646 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:23.884221 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:23.884223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:23.884224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:23.884287 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:23.884614 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:23.884615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:23.884616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:23.884761 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:23.884867 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:23.885286 (+ 419us) server_negotiation.cc:299] Negotiation successful
0807 08:47:23.885397 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 08:47:23.885780 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.873415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41913 (local address 127.0.0.1:51306)
0807 08:47:23.873812 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.873857 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:23.874527 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:23.874849 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:23.874853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:23.875323 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:23.875612 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:23.875619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.876785 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.876787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:23.878449 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:23.878453 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.879122 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.879123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:23.879257 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:23.879945 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:23.879956 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:23.881757 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:23.883778 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:23.883792 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:23.883795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:23.884106 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:23.884378 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:23.884379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:23.884379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:23.884531 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:23.884859 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:23.884869 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:23.885090 (+ 221us) client_negotiation.cc:769] Sending connection context
0807 08:47:23.885243 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 08:47:23.885464 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":156,"threads_started":1}
I20260807 08:47:23.886605 21655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.108.188@KRBTEST.COM" created.
Aug 07 08:47:23 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
I20260807 08:47:23.896096 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.886924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40751 (local address 127.0.0.1:57148)
0807 08:47:23.886994 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.887009 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:23.887061 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:23.887382 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:23.887383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:23.887536 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:23.887605 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:23.887606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.888724 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.888725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:23.889291 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:23.889293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.890046 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.890047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:23.890120 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:23.890691 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:23.890705 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:23.892190 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:23.894786 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:23.894786 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:23.894787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:23.894966 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:23.895237 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:23.895238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:23.895238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:23.895368 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:23.895809 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:23.895811 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:23.895859 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 08:47:23.895929 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 08:47:23.896019 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260807 08:47:23.896307 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.887012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.28.108.189:40751)
0807 08:47:23.887215 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:23.887220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:23.887249 (+ 29us) server_negotiation.cc:407] Connection header received
0807 08:47:23.887275 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:23.887277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:23.887289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:23.887354 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:23.887677 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.888621 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.889384 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.889946 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.890037 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:23.892360 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:23.892371 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:23.892371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:23.892378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:23.894661 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:23.895061 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:23.895066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:23.895067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:23.895146 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:23.895441 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:23.895442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:23.895443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:23.895695 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:23.895809 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:23.895999 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 08:47:23.896112 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":120,"threads_started":1}
I20260807 08:47:23.897128 21655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:23.897541 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.897415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:39584)
0807 08:47:23.897468 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.897518 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260807 08:47:23.897841 21600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
I20260807 08:47:23.898633 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.898508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:39592)
0807 08:47:23.898570 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.898609 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 08:47:23.898804 21655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:38965: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.108.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab.
Entry for principal kudu/127.28.108.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab.
I20260807 08:47:23.910491 29106 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskufh9Yd/build/release/bin/kudu
/tmp/dist-test-taskufh9Yd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.188:38965
--webserver_interface=127.28.108.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab
--principal=kudu/127.28.108.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.108.148:42133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.108.190:41913,127.28.108.189:40751,127.28.108.188:38965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 08:47:23.918985 21575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } attempt: 3
Aug 07 08:47:23 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
I20260807 08:47:23.927870 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.919592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.28.108.189:40751)
0807 08:47:23.919668 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:23.919671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:23.919684 (+ 13us) server_negotiation.cc:407] Connection header received
0807 08:47:23.919760 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:23.919761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:23.919778 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:23.919858 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:23.920230 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.920767 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.921549 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.921865 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.921950 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:23.924135 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:23.924136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:23.924136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:23.924144 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:23.926341 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:23.926860 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:23.926861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:23.926862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:23.926933 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:23.927186 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:23.927188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:23.927188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:23.927424 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:23.927495 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:23.927686 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 08:47:23.927796 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 08:47:23.927909 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.919506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40751 (local address 127.0.0.1:57152)
0807 08:47:23.919594 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.919610 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:23.919688 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:23.919868 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:23.919869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:23.920058 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:23.920134 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:23.920136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.920895 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.920896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:23.921452 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:23.921454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:23.921950 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:23.921951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:23.922039 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:23.922550 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:23.922555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:23.923988 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:23.926442 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:23.926444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:23.926444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:23.926739 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:23.927018 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:23.927019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:23.927020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:23.927093 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:23.927502 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:23.927503 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:23.927566 (+ 63us) client_negotiation.cc:769] Sending connection context
0807 08:47:23.927640 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 08:47:23.927760 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 08:47:23.928509 21575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:23.928910 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.928788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:39594)
0807 08:47:23.928837 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.928877 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260807 08:47:23.929260 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.929163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:39610)
0807 08:47:23.929205 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.929233 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260807 08:47:23.929368 21575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:38965: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
I20260807 08:47:23.948096 21655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } attempt: 1
I20260807 08:47:23.948663 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.948515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:39626)
0807 08:47:23.948587 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.948631 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 08:47:23.949055 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.948952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:39638)
0807 08:47:23.949002 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.949035 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260807 08:47:23.949152 21655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:38965: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
I20260807 08:47:23.960551 21575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } attempt: 1
I20260807 08:47:23.961167 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.960989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:39650)
0807 08:47:23.961063 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.961114 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260807 08:47:23.961650 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:23.961516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:39664)
0807 08:47:23.961573 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:23.961607 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260807 08:47:23.961771 21575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:38965: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
W20260807 08:47:24.002422 21662 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 08:47:24.002677 21662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 08:47:24.002696 21662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 08:47:24.002723 21662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 08:47:24.004082 21662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 08:47:24.004145 21662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 08:47:24.004160 21662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 08:47:24.004173 21662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 08:47:24.004186 21662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 08:47:24.006021 21662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.108.148:42133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.108.190:41913,127.28.108.189:40751,127.28.108.188:38965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/krb5kdc/kudu_127.28.108.188.keytab
--principal=kudu/127.28.108.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.108.188:38965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.108.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 07:54:20 UTC on ccycloud.quasar-zmatsq.root.comops.site
build id 4572006
I20260807 08:47:24.006342 21662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 08:47:24.006611 21662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 08:47:24.012264 21668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 08:47:24.012362 21667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:24.012542 21662 server_base.cc:1034] running on GCE node
W20260807 08:47:24.012574 21671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 08:47:24.012909 21662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 08:47:24.013242 21662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 08:47:24.014459 21662 hybrid_clock.cc:648] HybridClock initialized: now 1786092444014430 us; error 44 us; skew 500 ppm
Aug 07 08:47:24 dist-test-slave-9zdj krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092444, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 08:47:24.017330 21655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } attempt: 2
I20260807 08:47:24.017858 21662 init.cc:377] Logged in from keytab as kudu/127.28.108.188@KRBTEST.COM (short username kudu)
I20260807 08:47:24.018091 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.017686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:39672)
0807 08:47:24.017968 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:24.018046 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 08:47:24.018529 21662 webserver.cc:460] Webserver started at http://127.28.108.188:41017/ using document root <none> and password file <none>
I20260807 08:47:24.018821 21662 fs_manager.cc:359] Metadata directory not provided
I20260807 08:47:24.018888 21662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 08:47:24.018872 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.018518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:52610)
0807 08:47:24.018697 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:24.018799 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":247,"threads_started":1}
I20260807 08:47:24.019047 21662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 08:47:24.020398 21662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/data/instance:
uuid: "089e7aaab6f44b5b9e8147c1fb46a209"
format_stamp: "Formatted at 2026-08-07 08:47:24 on dist-test-slave-9zdj"
server_key: "6f8d3dc18c529e570a1ec14529215980"
server_key_iv: "21b0f397057192d539151aa25acd3c5f"
server_key_version: "encryptionkey@0"
W20260807 08:47:24.019068 21655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:38965: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
I20260807 08:47:24.020843 21662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/wal/instance:
uuid: "089e7aaab6f44b5b9e8147c1fb46a209"
format_stamp: "Formatted at 2026-08-07 08:47:24 on dist-test-slave-9zdj"
server_key: "6f8d3dc18c529e570a1ec14529215980"
server_key_iv: "21b0f397057192d539151aa25acd3c5f"
server_key_version: "encryptionkey@0"
I20260807 08:47:24.023619 21662 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.004s
I20260807 08:47:24.025036 21679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 08:47:24.025318 21662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 08:47:24.025475 21662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/data,/tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/wal
uuid: "089e7aaab6f44b5b9e8147c1fb46a209"
format_stamp: "Formatted at 2026-08-07 08:47:24 on dist-test-slave-9zdj"
server_key: "6f8d3dc18c529e570a1ec14529215980"
server_key_iv: "21b0f397057192d539151aa25acd3c5f"
server_key_version: "encryptionkey@0"
I20260807 08:47:24.025599 21662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 08:47:24.036937 21575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } attempt: 2
I20260807 08:47:24.037562 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.037422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:52616)
0807 08:47:24.037476 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:24.037523 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260807 08:47:24.038314 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.038086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:52626)
0807 08:47:24.038153 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:24.038191 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 08:47:24.038508 21575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.108.188:38965: Network error: Client connection negotiation failed: client connection to 127.28.108.188:38965: connect: Connection refused (error 111)
I20260807 08:47:24.042232 21662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 08:47:24.045126 21662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 08:47:24.045328 21662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 08:47:24.050057 21662 rpc_server.cc:307] RPC server started. Bound to: 127.28.108.188:38965
I20260807 08:47:24.050123 21738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.108.188:38965 every 8 connection(s)
I20260807 08:47:24.050578 21662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/data/info.pb
I20260807 08:47:24.051622 21739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 08:47:24.053442 21739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:24.057942 29106 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskufh9Yd/build/release/bin/kudu as pid 21662
I20260807 08:47:24.058096 29106 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskufh9Yd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786092133523677-29106-0/minicluster-data/master-2/wal/instance
I20260807 08:47:24.058367 29106 external_mini_cluster.cc:1428] Setting key 45a717eba678b47d2034eb6f030b73aa
Aug 07 08:47:24 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092444, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
Aug 07 08:47:24 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
I20260807 08:47:24.065435 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.054426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42522 (local address 127.28.108.190:41913)
0807 08:47:24.054609 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.054612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.054816 (+ 204us) server_negotiation.cc:407] Connection header received
0807 08:47:24.055453 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.055454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.055467 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.055532 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.056258 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.056761 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.057893 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.058128 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.058195 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.061196 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:24.061198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:24.061198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:24.061209 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:24.063600 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.064186 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.064188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.064188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.064267 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.064613 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.064614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.064614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.064782 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:24.064856 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:24.065234 (+ 378us) server_negotiation.cc:299] Negotiation successful
0807 08:47:24.065333 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260807 08:47:24.065717 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.054267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.190:41913 (local address 127.0.0.1:42522)
0807 08:47:24.054638 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:24.054689 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:24.055246 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:24.055617 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:24.055621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:24.055930 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:24.056152 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:24.056157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.056878 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.056879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:24.057799 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:24.057802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.058215 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.058216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:24.058358 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.059054 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:24.059079 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:24.061086 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:24.063717 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:24.063727 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:24.063737 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:24.064072 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:24.064386 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:24.064387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:24.064387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:24.064523 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:24.064864 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:24.064874 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:24.065034 (+ 160us) client_negotiation.cc:769] Sending connection context
0807 08:47:24.065192 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 08:47:24.065376 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":131,"threads_started":1}
I20260807 08:47:24.066851 21739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:24.069875 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.058893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.28.108.188:38965)
0807 08:47:24.059131 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.059136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.059220 (+ 84us) server_negotiation.cc:407] Connection header received
0807 08:47:24.059253 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.059256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.059287 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.059377 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.060227 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.061199 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.062331 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.062849 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.062918 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.066121 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:24.066133 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:24.066134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:24.066142 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:24.068323 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.068750 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.068754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.068754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.068829 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.069059 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.069059 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.069060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.069213 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:24.069355 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:24.069540 (+ 185us) server_negotiation.cc:299] Negotiation successful
0807 08:47:24.069668 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":112,"threads_started":1}
Aug 07 08:47:24 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092444, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
I20260807 08:47:24.075418 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.067506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.28.108.189:40751)
0807 08:47:24.067583 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.067585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.067602 (+ 17us) server_negotiation.cc:407] Connection header received
0807 08:47:24.067631 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.067632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.067644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.067714 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.068106 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.068813 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.069888 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.070352 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.070420 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.072392 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:24.072393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:24.072394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:24.072403 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:24.074200 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.074592 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.074593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.074594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.074662 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.074942 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.074943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.074943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.075062 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:24.075120 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:24.075284 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 08:47:24.075347 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Aug 07 08:47:24 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.190@KRBTEST.COM
I20260807 08:47:24.075487 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.067311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.189:40751 (local address 127.0.0.1:33044)
0807 08:47:24.067464 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:24.067477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:24.067520 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:24.067805 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:24.067806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:24.067952 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:24.068000 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:24.068009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.068914 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.068915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:24.069814 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:24.069818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.070434 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.070435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:24.070514 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.071032 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:24.071038 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:24.072286 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:24.074316 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:24.074317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:24.074318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:24.074511 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:24.074733 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:24.074734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:24.074734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:24.074875 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:24.075133 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:24.075134 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:24.075191 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 08:47:24.075270 (+ 79us) client_negotiation.cc:240] Negotiation successful
0807 08:47:24.075382 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260807 08:47:24.076045 21739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } has no permanent_uuid. Determining permanent_uuid...
I20260807 08:47:24.079432 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.071004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.28.108.190:41913)
0807 08:47:24.071055 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.071057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.071063 (+ 6us) server_negotiation.cc:407] Connection header received
0807 08:47:24.071089 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.071089 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.071104 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.071170 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.071701 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.072266 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.073168 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.073382 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.073460 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.076014 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:24.076016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:24.076016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:24.076025 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:24.077935 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.078435 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.078436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.078436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.078498 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.078719 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.078720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.078720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.078838 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:24.079071 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:24.079110 (+ 39us) server_negotiation.cc:299] Negotiation successful
0807 08:47:24.079201 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Aug 07 08:47:24 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092444, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.188@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
I20260807 08:47:24.084617 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.076362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:52634)
0807 08:47:24.076600 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:24.076612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:24.076656 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:24.076829 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:24.076829 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:24.077012 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:24.077063 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:24.077065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.078077 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.078078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:24.078886 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:24.078889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.079423 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.079423 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:24.079476 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.079905 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:24.079912 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:24.081591 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:24.083403 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:24.083404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:24.083405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:24.083584 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:24.083819 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:24.083820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:24.083820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:24.083887 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:24.084149 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:24.084150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:24.084192 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 08:47:24.084284 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 08:47:24.084426 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":3}
I20260807 08:47:24.084625 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.076397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.28.108.188:38965)
0807 08:47:24.076462 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.076465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.076666 (+ 201us) server_negotiation.cc:407] Connection header received
0807 08:47:24.076694 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.076695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.076711 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.076829 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.077165 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.077974 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.078978 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.079298 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.079373 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.081751 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:24.081753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:24.081754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:24.081762 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:24.083296 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.083674 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.083675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.083675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.083736 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.083958 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.083959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.083959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.084081 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:24.084159 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:24.084310 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 08:47:24.084525 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":40}
Aug 07 08:47:24 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.108.189@KRBTEST.COM
I20260807 08:47:24.087177 21739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209: Bootstrap starting.
I20260807 08:47:24.088449 21739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:24.088801 21739 log.cc:824] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:24.090030 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.080149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.28.108.189:40751)
0807 08:47:24.080229 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.080231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.080241 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:47:24.080269 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.080270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.080375 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.080450 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.081205 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.081884 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.082888 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.083632 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.083773 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.086598 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:24.086600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:24.086601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:24.086611 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:24.088520 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.089011 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.089013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.089013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.089110 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.089328 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.089330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.089330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.089504 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:24.089754 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:24.089825 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 08:47:24.089921 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 08:47:24.090023 21739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209: No bootstrap required, opened a new log
I20260807 08:47:24.092006 29106 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 08:47:24.092397 21739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } }
I20260807 08:47:24.092545 21739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089e7aaab6f44b5b9e8147c1fb46a209, State: Initialized, Role: FOLLOWER
I20260807 08:47:24.092638 21739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } }
I20260807 08:47:24.093047 21739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:24.093001 21745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } } }
I20260807 08:47:24.093135 21745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:47:24.093290 21546 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f33dbf21d611473dac4245d89acf9558"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42538
W20260807 08:47:24.095794 21757 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:47:24.095893 21757 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:47:24.095767 21662 master_runner.cc:186] Error getting master registration for 127.28.108.190:41913: OK, instance_id { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" instance_seqno: 1786092443685195 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 08:47:24.098521 21662 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 08:47:24.111954 21655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } attempt: 3
Aug 07 08:47:24 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.189@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
I20260807 08:47:24.122318 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.112670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.28.108.188:38965)
0807 08:47:24.112742 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.112745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.112798 (+ 53us) server_negotiation.cc:407] Connection header received
0807 08:47:24.112929 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.112931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.112943 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.113028 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.113652 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.114618 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.115613 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.115858 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.115969 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.118565 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:24.118567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:24.118568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:24.118580 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:24.120628 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.121159 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.121161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.121161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.121243 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.121488 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.121489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.121489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.121795 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:24.121894 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:24.122093 (+ 199us) server_negotiation.cc:299] Negotiation successful
0807 08:47:24.122202 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 08:47:24.122349 21677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.112602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:52640)
0807 08:47:24.112720 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:24.112736 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:24.112822 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:24.113072 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:24.113073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:24.113367 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:24.113483 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:24.113485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.114741 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.114742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:24.115487 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:24.115490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.115973 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.115975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:24.116072 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.116678 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:24.116684 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:24.118413 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:24.120763 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:24.120765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:24.120765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:24.121058 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:24.121331 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:24.121332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:24.121332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:24.121410 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:24.121894 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:24.121895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:24.121960 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 08:47:24.122045 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 08:47:24.122188 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 08:47:24.124049 21655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60: Bootstrap starting.
I20260807 08:47:24.124818 21655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:24.125227 21655 log.cc:824] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:24.126178 21655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60: No bootstrap required, opened a new log
I20260807 08:47:24.127841 21655 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } }
I20260807 08:47:24.128016 21655 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8455ceb81d3428e988d8336bedcaa60, State: Initialized, Role: FOLLOWER
I20260807 08:47:24.128167 21655 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } }
I20260807 08:47:24.128611 21760 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } } }
I20260807 08:47:24.128705 21655 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:24.128759 21760 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 08:47:24.131685 21774 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:47:24.131772 21774 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:47:24.131826 21582 master_runner.cc:186] Error getting master registration for 127.28.108.190:41913: OK, instance_id { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" instance_seqno: 1786092443685195 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 08:47:24.133738 21582 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 08:47:24.137735 21575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } attempt: 3
Aug 07 08:47:24 dist-test-slave-9zdj krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786092443, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.108.190@KRBTEST.COM for kudu/127.28.108.188@KRBTEST.COM
I20260807 08:47:24.148142 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.138166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.108.188:38965 (local address 127.0.0.1:52648)
0807 08:47:24.138256 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0807 08:47:24.138273 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 08:47:24.138343 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 08:47:24.138553 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 08:47:24.138554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 08:47:24.138782 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0807 08:47:24.138856 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:24.138859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.139994 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.139995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:24.140930 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 08:47:24.140934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.141503 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.141505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 08:47:24.141575 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.142185 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 08:47:24.142192 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 08:47:24.144309 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 08:47:24.146720 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:24.146722 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:24.146723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:24.147019 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:24.147285 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 08:47:24.147286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 08:47:24.147287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 08:47:24.147370 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 08:47:24.147718 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 08:47:24.147719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 08:47:24.147783 (+ 64us) client_negotiation.cc:769] Sending connection context
0807 08:47:24.147874 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 08:47:24.148010 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 08:47:24.148150 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.138326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.28.108.188:38965)
0807 08:47:24.138392 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.138395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.138406 (+ 11us) server_negotiation.cc:407] Connection header received
0807 08:47:24.138440 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.138441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.138453 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.138527 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.138994 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.139874 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.141059 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.141406 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.141488 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.144426 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:24.144428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:24.144428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:24.144440 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:24.146602 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.147107 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.147109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.147109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.147189 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.147446 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.147447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.147448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.147633 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:24.147717 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:24.147927 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 08:47:24.148043 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 08:47:24.150643 21575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558: Bootstrap starting.
I20260807 08:47:24.151559 21575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558: Neither blocks nor log segments found. Creating new log.
I20260807 08:47:24.151966 21575 log.cc:824] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558: Log is configured to *not* fsync() on all Append() calls
I20260807 08:47:24.153257 21575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558: No bootstrap required, opened a new log
I20260807 08:47:24.155066 21575 raft_consensus.cc:348] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } }
I20260807 08:47:24.155259 21575 raft_consensus.cc:729] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33dbf21d611473dac4245d89acf9558, State: Initialized, Role: FOLLOWER
I20260807 08:47:24.155359 21575 consensus_queue.cc:260] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } }
I20260807 08:47:24.155771 21575 sys_catalog.cc:564] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 08:47:24.155827 21546 raft_consensus.cc:482] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 08:47:24.155917 21546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:24.156946 21546 raft_consensus.cc:504] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } }
I20260807 08:47:24.157299 21546 leader_election.cc:288] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [CANDIDATE]: Term 1 election: Requested vote from peers b8455ceb81d3428e988d8336bedcaa60 (127.28.108.189:40751), 089e7aaab6f44b5b9e8147c1fb46a209 (127.28.108.188:38965)
I20260807 08:47:24.157850 21628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f33dbf21d611473dac4245d89acf9558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b8455ceb81d3428e988d8336bedcaa60"
I20260807 08:47:24.158032 21628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:24.159163 21628 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f33dbf21d611473dac4245d89acf9558 in term 1.
I20260807 08:47:24.159505 21706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f33dbf21d611473dac4245d89acf9558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "089e7aaab6f44b5b9e8147c1fb46a209"
I20260807 08:47:24.159667 21706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [term 0 FOLLOWER]: Advancing to term 1
I20260807 08:47:24.160730 21706 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f33dbf21d611473dac4245d89acf9558 in term 1.
I20260807 08:47:24.161113 21518 leader_election.cc:302] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 089e7aaab6f44b5b9e8147c1fb46a209, f33dbf21d611473dac4245d89acf9558; no voters:
I20260807 08:47:24.155774 21777 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } } }
I20260807 08:47:24.161526 21777 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:47:24.161592 21777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 08:47:24.161696 21777 raft_consensus.cc:686] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [term 1 LEADER]: Becoming Leader. State: Replica: f33dbf21d611473dac4245d89acf9558, State: Running, Role: LEADER
I20260807 08:47:24.161795 21777 consensus_queue.cc:237] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } }
I20260807 08:47:24.163252 21777 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f33dbf21d611473dac4245d89acf9558. Latest consensus state: current_term: 1 leader_uuid: "f33dbf21d611473dac4245d89acf9558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } } }
I20260807 08:47:24.163364 21777 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [sys.catalog]: This master's current role is: LEADER
W20260807 08:47:24.164568 21789 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 08:47:24.164645 21789 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 08:47:24.165311 21798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 08:47:24.165545 21798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 08:47:24.170990 21706 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [term 1 FOLLOWER]: Refusing update from remote peer f33dbf21d611473dac4245d89acf9558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:47:24.171821 21628 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [term 1 FOLLOWER]: Refusing update from remote peer f33dbf21d611473dac4245d89acf9558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 08:47:24.172133 21777 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:47:24.173759 21745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f33dbf21d611473dac4245d89acf9558. Latest consensus state: current_term: 1 leader_uuid: "f33dbf21d611473dac4245d89acf9558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } } }
I20260807 08:47:24.173969 21745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:47:24.174052 21777 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 08:47:24.175516 21760 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f33dbf21d611473dac4245d89acf9558. Latest consensus state: current_term: 1 leader_uuid: "f33dbf21d611473dac4245d89acf9558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } } }
I20260807 08:47:24.175635 21760 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:47:24.180238 21777 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f33dbf21d611473dac4245d89acf9558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } } }
I20260807 08:47:24.180369 21777 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:24.180790 21745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f33dbf21d611473dac4245d89acf9558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } } }
I20260807 08:47:24.180912 21745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:47:24.181445 21801 mvcc.cc:205] Tried to move back new op lower bound from 7315834651321090048 to 7315834651287851008. Current Snapshot: MvccSnapshot[applied={T|T < 7315834651321090048}]
I20260807 08:47:24.182716 21804 mvcc.cc:205] Tried to move back new op lower bound from 7315834651321090048 to 7315834651287851008. Current Snapshot: MvccSnapshot[applied={T|T < 7315834651321090048}]
I20260807 08:47:24.183503 21777 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f33dbf21d611473dac4245d89acf9558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } } }
I20260807 08:47:24.183616 21777 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558 [sys.catalog]: This master's current role is: LEADER
I20260807 08:47:24.184173 21760 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f33dbf21d611473dac4245d89acf9558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33dbf21d611473dac4245d89acf9558" member_type: VOTER last_known_addr { host: "127.28.108.190" port: 41913 } } peers { permanent_uuid: "b8455ceb81d3428e988d8336bedcaa60" member_type: VOTER last_known_addr { host: "127.28.108.189" port: 40751 } } peers { permanent_uuid: "089e7aaab6f44b5b9e8147c1fb46a209" member_type: VOTER last_known_addr { host: "127.28.108.188" port: 38965 } } }
I20260807 08:47:24.184286 21760 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 08:47:24.184489 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.159226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.28.108.189:40751)
0807 08:47:24.159282 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.159285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.169890 (+ 10605us) server_negotiation.cc:407] Connection header received
0807 08:47:24.170030 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.170032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.170047 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.170083 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.171301 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.172025 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.175387 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.175697 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.175744 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.178125 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:24.178127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:24.178127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:24.178136 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:24.179802 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.181157 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.181159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.181159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.181222 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.181872 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.181873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.181873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.182018 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:24.182102 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:24.184296 (+ 2194us) server_negotiation.cc:299] Negotiation successful
0807 08:47:24.184404 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 08:47:24.184553 21806 mvcc.cc:205] Tried to move back new op lower bound from 7315834651321090048 to 7315834651287851008. Current Snapshot: MvccSnapshot[applied={T|T < 7315834651321090048}]
I20260807 08:47:24.185509 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.159225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.28.108.190:41913)
0807 08:47:24.159323 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.159325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.170414 (+ 11089us) server_negotiation.cc:407] Connection header received
0807 08:47:24.170542 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.170543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.170556 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.170587 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.173540 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.174285 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.176143 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.176572 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.176631 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.180850 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:24.180852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:24.180852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:24.180861 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:24.182807 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.183559 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.183560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.183561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.183626 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.183843 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.183844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.183844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.183980 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:24.184054 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:24.185320 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0807 08:47:24.185415 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 08:47:24.190272 21798 catalog_manager.cc:1347] Generated new cluster ID: 86aa5ea6dd6b4991a0e6687263b4c0d1
I20260807 08:47:24.190366 21798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 08:47:24.198628 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:24.170015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.28.108.188:38965)
0807 08:47:24.170067 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.170069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.178494 (+ 8425us) server_negotiation.cc:407] Connection header received
0807 08:47:24.178624 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.178626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.178640 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.178677 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.179607 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.180321 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.182541 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:24.182868 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:24.182923 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:24.195116 (+ 12193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 08:47:24.195118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 08:47:24.195119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 08:47:24.195129 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 08:47:24.196970 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.197443 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.197444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.197444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.197503 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 08:47:24.197719 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 08:47:24.197720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 08:47:24.197720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 08:47:24.197866 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 08:47:24.197951 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:24.198441 (+ 490us) server_negotiation.cc:299] Negotiation successful
0807 08:47:24.198520 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 08:47:24.213601 21798 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 08:47:24.214515 21798 catalog_manager.cc:1504] Loading token signing keys...
I20260807 08:47:24.222018 21798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f33dbf21d611473dac4245d89acf9558: Generated new TSK 0
I20260807 08:47:24.222301 21798 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 08:47:24.255704 21578 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.253983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.28.108.190:41913)
0807 08:47:24.254039 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.254041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.254853 (+ 812us) server_negotiation.cc:407] Connection header received
0807 08:47:24.254978 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.254979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.254992 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.255024 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.255600 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42562: BlockingRecv error: recv got EOF from 127.0.0.1:42562 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260807 08:47:24.256855 21743 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.254522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.28.108.188:38965)
0807 08:47:24.254592 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.254595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.256190 (+ 1595us) server_negotiation.cc:407] Connection header received
0807 08:47:24.256313 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.256314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.256329 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.256365 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.256775 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52660: BlockingRecv error: recv got EOF from 127.0.0.1:52660 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260807 08:47:24.258285 21659 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.257457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.28.108.189:40751)
0807 08:47:24.257501 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.257503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.257681 (+ 178us) server_negotiation.cc:407] Connection header received
0807 08:47:24.257787 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.257788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.257799 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.257831 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.258217 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33070: BlockingRecv error: recv got EOF from 127.0.0.1:33070 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260807 08:47:24.261168 21659 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.260269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.28.108.189:40751)
0807 08:47:24.260327 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.260329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.260543 (+ 214us) server_negotiation.cc:407] Connection header received
0807 08:47:24.260663 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.260664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.260686 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.260718 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.261110 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33072: BlockingRecv error: recv got EOF from 127.0.0.1:33072 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260807 08:47:24.265318 21659 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.264274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.28.108.189:40751)
0807 08:47:24.264334 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.264336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.264635 (+ 299us) server_negotiation.cc:407] Connection header received
0807 08:47:24.264749 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.264750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.264763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.264795 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.265244 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33080: BlockingRecv error: recv got EOF from 127.0.0.1:33080 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260807 08:47:24.270017 21659 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.268958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.28.108.189:40751)
0807 08:47:24.269005 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.269008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.269268 (+ 260us) server_negotiation.cc:407] Connection header received
0807 08:47:24.269400 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.269401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.269414 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.269447 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.269936 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33084: BlockingRecv error: recv got EOF from 127.0.0.1:33084 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260807 08:47:24.276779 21659 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.275741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.28.108.189:40751)
0807 08:47:24.275788 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.275790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.276067 (+ 277us) server_negotiation.cc:407] Connection header received
0807 08:47:24.276181 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.276182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.276194 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.276226 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.276714 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33096: BlockingRecv error: recv got EOF from 127.0.0.1:33096 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260807 08:47:24.287954 21659 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.286640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.28.108.189:40751)
0807 08:47:24.286697 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.286700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.287006 (+ 306us) server_negotiation.cc:407] Connection header received
0807 08:47:24.287126 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.287127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.287140 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.287172 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.287884 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33110: BlockingRecv error: recv got EOF from 127.0.0.1:33110 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260807 08:47:24.306828 21659 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.305778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.28.108.189:40751)
0807 08:47:24.305827 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.305829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.306132 (+ 303us) server_negotiation.cc:407] Connection header received
0807 08:47:24.306246 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.306248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.306260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.306292 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.306765 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33120: BlockingRecv error: recv got EOF from 127.0.0.1:33120 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260807 08:47:24.341813 21659 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.340810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.28.108.189:40751)
0807 08:47:24.340899 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.340902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.340966 (+ 64us) server_negotiation.cc:407] Connection header received
0807 08:47:24.341079 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.341080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.341093 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.341196 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.341731 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33132: BlockingRecv error: recv got EOF from 127.0.0.1:33132 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260807 08:47:24.408147 21659 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.407255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.28.108.189:40751)
0807 08:47:24.407336 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.407338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.407470 (+ 132us) server_negotiation.cc:407] Connection header received
0807 08:47:24.407531 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.407532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.407544 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.407591 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.408085 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33136: BlockingRecv error: recv got EOF from 127.0.0.1:33136 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260807 08:47:24.538511 21659 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.537705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.28.108.189:40751)
0807 08:47:24.537791 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.537794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.537874 (+ 80us) server_negotiation.cc:407] Connection header received
0807 08:47:24.537932 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.537933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.537961 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.538041 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.538458 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33152: BlockingRecv error: recv got EOF from 127.0.0.1:33152 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260807 08:47:24.797473 21659 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 08:47:24.796378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.28.108.189:40751)
0807 08:47:24.796505 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:24.796509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:24.796827 (+ 318us) server_negotiation.cc:407] Connection header received
0807 08:47:24.796869 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:24.796870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:24.796886 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:24.797024 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 08:47:24.797420 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33156: BlockingRecv error: recv got EOF from 127.0.0.1:33156 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 08:47:25.096547 21757 catalog_manager.cc:1259] Loaded cluster ID: 86aa5ea6dd6b4991a0e6687263b4c0d1
I20260807 08:47:25.096647 21757 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209: loading cluster ID for follower catalog manager: success
I20260807 08:47:25.097692 21757 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209: acquiring CA information for follower catalog manager: success
I20260807 08:47:25.098104 21757 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 089e7aaab6f44b5b9e8147c1fb46a209: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 08:47:25.132287 21774 catalog_manager.cc:1259] Loaded cluster ID: 86aa5ea6dd6b4991a0e6687263b4c0d1
I20260807 08:47:25.132392 21774 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60: loading cluster ID for follower catalog manager: success
I20260807 08:47:25.133358 21774 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60: acquiring CA information for follower catalog manager: success
I20260807 08:47:25.133777 21774 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8455ceb81d3428e988d8336bedcaa60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 08:47:25.314404 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:25.311036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.28.108.189:40751)
0807 08:47:25.311194 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:25.311197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:25.311352 (+ 155us) server_negotiation.cc:407] Connection header received
0807 08:47:25.311455 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:25.311458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:25.311527 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:25.311634 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:47:25.311817 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:25.312445 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:25.313390 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:25.313674 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:25.313759 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:25.313984 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:47:25.314121 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:47:25.314225 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:25.314251 (+ 26us) server_negotiation.cc:299] Negotiation successful
0807 08:47:25.314281 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260807 08:47:25.319701 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:25.316633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.28.108.188:38965)
0807 08:47:25.316768 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:25.316770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:25.316780 (+ 10us) server_negotiation.cc:407] Connection header received
0807 08:47:25.316870 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:25.316871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:25.316915 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:25.317004 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:47:25.317108 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:25.317765 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:25.318722 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:25.318998 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:25.319066 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:25.319362 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:47:25.319484 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:47:25.319550 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:25.319601 (+ 51us) server_negotiation.cc:299] Negotiation successful
0807 08:47:25.319624 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260807 08:47:25.325737 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:25.322144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.28.108.189:40751)
0807 08:47:25.322398 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:25.322401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:25.322669 (+ 268us) server_negotiation.cc:407] Connection header received
0807 08:47:25.322744 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:25.322745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:25.322758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:25.322811 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:47:25.322915 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:25.323467 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:25.324822 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:25.325147 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:25.325216 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:25.325407 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:47:25.325448 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:47:25.325497 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:25.325596 (+ 99us) server_negotiation.cc:299] Negotiation successful
0807 08:47:25.325654 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260807 08:47:25.325726 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:25.322417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.28.108.188:38965)
0807 08:47:25.322482 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:25.322484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:25.322496 (+ 12us) server_negotiation.cc:407] Connection header received
0807 08:47:25.322541 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:25.322542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:25.322555 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:25.322614 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:47:25.322854 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:25.323640 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:25.324776 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:25.325074 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:25.325184 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:25.325346 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:47:25.325389 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:47:25.325438 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:25.325554 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 08:47:25.325586 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260807 08:47:25.326371 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 08:47:25.322213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.28.108.190:41913)
0807 08:47:25.322398 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 08:47:25.322401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 08:47:25.322567 (+ 166us) server_negotiation.cc:407] Connection header received
0807 08:47:25.322645 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 08:47:25.322646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 08:47:25.322717 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 08:47:25.322806 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 08:47:25.323025 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:25.323806 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:25.324620 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 08:47:25.324931 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 08:47:25.325003 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 08:47:25.325282 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 08:47:25.325430 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 08:47:25.325508 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 08:47:25.326269 (+ 761us) server_negotiation.cc:299] Negotiation successful
0807 08:47:25.326297 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":122,"threads_started":1}
I20260807 08:47:25.336222 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 21501
I20260807 08:47:25.345777 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 21582
I20260807 08:47:25.354183 29106 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskufh9Yd/build/release/bin/kudu with pid 21662
2026-08-07T08:47:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3125 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6293 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (311823 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST